2023-02-28 04:56:59,428 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290 2023-02-28 04:56:59,440 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple timeout: 13 mins 2023-02-28 04:56:59,454 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-28 04:56:59,454 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/cluster_b26b0118-f74c-2753-b135-766a0fc054b5, deleteOnExit=true 2023-02-28 04:56:59,454 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-28 04:56:59,455 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/test.cache.data in system properties and HBase conf 2023-02-28 04:56:59,455 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/hadoop.tmp.dir in system properties and HBase conf 2023-02-28 04:56:59,455 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/hadoop.log.dir in system properties and HBase conf 2023-02-28 04:56:59,456 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-28 04:56:59,456 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-28 04:56:59,456 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-28 04:56:59,572 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-28 04:56:59,962 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-28 04:56:59,966 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-28 04:56:59,966 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-28 04:56:59,967 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-28 04:56:59,967 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-28 04:56:59,967 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-28 04:56:59,968 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-28 04:56:59,968 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-28 04:56:59,968 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-28 04:56:59,969 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-28 04:56:59,969 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/nfs.dump.dir in system properties and HBase conf 2023-02-28 04:56:59,970 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/java.io.tmpdir in system properties and HBase conf 2023-02-28 04:56:59,970 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-28 04:56:59,970 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-28 04:56:59,971 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-28 04:57:00,488 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-28 04:57:00,492 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-28 04:57:00,770 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-28 04:57:00,937 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-28 04:57:00,951 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-28 04:57:00,986 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-28 04:57:01,018 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/java.io.tmpdir/Jetty_localhost_45601_hdfs____.awtfwg/webapp 2023-02-28 04:57:01,175 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45601 2023-02-28 04:57:01,186 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-28 04:57:01,187 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-28 04:57:01,561 WARN [Listener at localhost/33733] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-28 04:57:01,660 WARN [Listener at localhost/33733] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-28 04:57:01,677 WARN [Listener at localhost/33733] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-28 04:57:01,685 INFO [Listener at localhost/33733] log.Slf4jLog(67): jetty-6.1.26 2023-02-28 04:57:01,690 INFO [Listener at localhost/33733] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/java.io.tmpdir/Jetty_localhost_42449_datanode____iv6voi/webapp 2023-02-28 04:57:01,791 INFO [Listener at localhost/33733] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42449 2023-02-28 04:57:02,108 WARN [Listener at localhost/45417] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-28 04:57:02,121 WARN [Listener at localhost/45417] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-28 04:57:02,123 WARN [Listener at localhost/45417] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-28 04:57:02,125 INFO [Listener at localhost/45417] log.Slf4jLog(67): jetty-6.1.26 2023-02-28 04:57:02,130 INFO [Listener at localhost/45417] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/java.io.tmpdir/Jetty_localhost_44611_datanode____.qbdcjn/webapp 2023-02-28 04:57:02,226 INFO [Listener at localhost/45417] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44611 2023-02-28 04:57:02,235 WARN [Listener at localhost/42809] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-28 04:57:02,247 WARN [Listener at localhost/42809] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-28 04:57:02,250 WARN [Listener at localhost/42809] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-28 04:57:02,252 INFO [Listener at localhost/42809] log.Slf4jLog(67): jetty-6.1.26 2023-02-28 04:57:02,259 INFO [Listener at localhost/42809] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/java.io.tmpdir/Jetty_localhost_36233_datanode____lcqcdy/webapp 2023-02-28 04:57:02,371 INFO [Listener at localhost/42809] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36233 2023-02-28 04:57:02,380 WARN [Listener at localhost/36897] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-28 04:57:02,591 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xab0e8f5715f6c252: Processing first storage report for DS-399135da-b37d-4a80-ae3b-3bbfc6e63945 from datanode ad0147c2-ff5d-4955-80a8-0012b47fa37a 2023-02-28 04:57:02,593 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xab0e8f5715f6c252: from storage DS-399135da-b37d-4a80-ae3b-3bbfc6e63945 node DatanodeRegistration(127.0.0.1:37967, datanodeUuid=ad0147c2-ff5d-4955-80a8-0012b47fa37a, infoPort=33047, infoSecurePort=0, ipcPort=45417, storageInfo=lv=-57;cid=testClusterID;nsid=225513654;c=1677560220563), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-02-28 04:57:02,593 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x72b80af9f2931d63: Processing first storage report for DS-9e76f0e4-3b34-423f-b2ed-93f1bf763d0e from datanode 8c9efb49-2a4b-4a87-bb8e-d6e1442f8c08 2023-02-28 04:57:02,593 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x72b80af9f2931d63: from storage DS-9e76f0e4-3b34-423f-b2ed-93f1bf763d0e node DatanodeRegistration(127.0.0.1:34137, datanodeUuid=8c9efb49-2a4b-4a87-bb8e-d6e1442f8c08, infoPort=38223, infoSecurePort=0, ipcPort=36897, storageInfo=lv=-57;cid=testClusterID;nsid=225513654;c=1677560220563), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-28 04:57:02,593 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x16e004c29a5609c: Processing first storage report for DS-994241a6-57fe-4042-bd49-62f8a7d0e6ea from datanode c977ee8b-e160-4115-8e8e-c9f50b5a113b 2023-02-28 04:57:02,594 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x16e004c29a5609c: from storage DS-994241a6-57fe-4042-bd49-62f8a7d0e6ea node DatanodeRegistration(127.0.0.1:37889, datanodeUuid=c977ee8b-e160-4115-8e8e-c9f50b5a113b, infoPort=40809, infoSecurePort=0, ipcPort=42809, storageInfo=lv=-57;cid=testClusterID;nsid=225513654;c=1677560220563), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-28 04:57:02,594 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xab0e8f5715f6c252: Processing first storage report for DS-bc3502e7-b0e8-4c5b-a3b8-c6d576c65e53 from datanode ad0147c2-ff5d-4955-80a8-0012b47fa37a 2023-02-28 04:57:02,594 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xab0e8f5715f6c252: from storage DS-bc3502e7-b0e8-4c5b-a3b8-c6d576c65e53 node DatanodeRegistration(127.0.0.1:37967, datanodeUuid=ad0147c2-ff5d-4955-80a8-0012b47fa37a, infoPort=33047, infoSecurePort=0, ipcPort=45417, storageInfo=lv=-57;cid=testClusterID;nsid=225513654;c=1677560220563), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-28 04:57:02,594 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x72b80af9f2931d63: Processing first storage report for DS-92ef5702-3fe3-47e2-a93e-27fedcca294c from datanode 8c9efb49-2a4b-4a87-bb8e-d6e1442f8c08 2023-02-28 04:57:02,594 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x72b80af9f2931d63: from storage DS-92ef5702-3fe3-47e2-a93e-27fedcca294c node DatanodeRegistration(127.0.0.1:34137, datanodeUuid=8c9efb49-2a4b-4a87-bb8e-d6e1442f8c08, infoPort=38223, infoSecurePort=0, ipcPort=36897, storageInfo=lv=-57;cid=testClusterID;nsid=225513654;c=1677560220563), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-28 04:57:02,594 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x16e004c29a5609c: Processing first storage report for DS-6a106a38-b58d-4ed1-80b6-3a590ff1d430 from datanode c977ee8b-e160-4115-8e8e-c9f50b5a113b 2023-02-28 04:57:02,595 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x16e004c29a5609c: from storage DS-6a106a38-b58d-4ed1-80b6-3a590ff1d430 node DatanodeRegistration(127.0.0.1:37889, datanodeUuid=c977ee8b-e160-4115-8e8e-c9f50b5a113b, infoPort=40809, infoSecurePort=0, ipcPort=42809, storageInfo=lv=-57;cid=testClusterID;nsid=225513654;c=1677560220563), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-02-28 04:57:02,760 DEBUG [Listener at localhost/36897] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290 2023-02-28 04:57:02,826 INFO [Listener at localhost/36897] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/cluster_b26b0118-f74c-2753-b135-766a0fc054b5/zookeeper_0, clientPort=54355, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/cluster_b26b0118-f74c-2753-b135-766a0fc054b5/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/cluster_b26b0118-f74c-2753-b135-766a0fc054b5/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-28 04:57:02,841 INFO [Listener at localhost/36897] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=54355 2023-02-28 04:57:02,851 INFO [Listener at localhost/36897] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-28 04:57:02,853 INFO [Listener at localhost/36897] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-28 04:57:03,518 INFO [Listener at localhost/36897] util.FSUtils(479): Created version file at hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f with version=8 2023-02-28 04:57:03,518 INFO [Listener at localhost/36897] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/hbase-staging 2023-02-28 04:57:03,828 INFO [Listener at localhost/36897] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-28 04:57:04,284 INFO [Listener at localhost/36897] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=18 2023-02-28 04:57:04,315 INFO [Listener at localhost/36897] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-28 04:57:04,316 INFO [Listener at localhost/36897] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-28 04:57:04,316 INFO [Listener at localhost/36897] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-28 04:57:04,317 INFO [Listener at localhost/36897] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-28 04:57:04,317 INFO [Listener at localhost/36897] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-28 04:57:04,464 INFO [Listener at localhost/36897] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-28 04:57:04,534 DEBUG [Listener at localhost/36897] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-28 04:57:04,625 INFO [Listener at localhost/36897] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42505 2023-02-28 04:57:04,635 INFO [Listener at localhost/36897] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-28 04:57:04,637 INFO [Listener at localhost/36897] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-28 04:57:04,659 INFO [Listener at localhost/36897] zookeeper.RecoverableZooKeeper(93): Process identifier=master:42505 connecting to ZooKeeper ensemble=127.0.0.1:54355 2023-02-28 04:57:04,697 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:425050x0, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-28 04:57:04,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:42505-0x10099497e1c0000 connected 2023-02-28 04:57:04,729 DEBUG [Listener at localhost/36897] zookeeper.ZKUtil(164): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-28 04:57:04,730 DEBUG [Listener at localhost/36897] zookeeper.ZKUtil(164): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-28 04:57:04,734 DEBUG [Listener at localhost/36897] zookeeper.ZKUtil(164): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-28 04:57:04,741 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42505 2023-02-28 04:57:04,741 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42505 2023-02-28 04:57:04,742 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42505 2023-02-28 04:57:04,742 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42505 2023-02-28 04:57:04,742 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42505 2023-02-28 04:57:04,748 INFO [Listener at localhost/36897] master.HMaster(444): hbase.rootdir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f, hbase.cluster.distributed=false 2023-02-28 04:57:04,814 INFO [Listener at localhost/36897] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-28 04:57:04,814 INFO [Listener at localhost/36897] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-28 04:57:04,814 INFO [Listener at localhost/36897] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-28 04:57:04,814 INFO [Listener at localhost/36897] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-28 04:57:04,815 INFO [Listener at localhost/36897] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-28 04:57:04,815 INFO [Listener at localhost/36897] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-28 04:57:04,819 INFO [Listener at localhost/36897] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-28 04:57:04,822 INFO [Listener at localhost/36897] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46629 2023-02-28 04:57:04,824 INFO [Listener at localhost/36897] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-28 04:57:04,829 INFO [Listener at localhost/36897] mob.MobFileCache(124): MobFileCache disabled 2023-02-28 04:57:04,831 INFO [Listener at localhost/36897] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-28 04:57:04,832 INFO [Listener at localhost/36897] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-28 04:57:04,834 INFO [Listener at localhost/36897] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:46629 connecting to ZooKeeper ensemble=127.0.0.1:54355 2023-02-28 04:57:04,838 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:466290x0, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-28 04:57:04,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:46629-0x10099497e1c0001 connected 2023-02-28 04:57:04,840 DEBUG [Listener at localhost/36897] zookeeper.ZKUtil(164): regionserver:46629-0x10099497e1c0001, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-28 04:57:04,841 DEBUG [Listener at localhost/36897] zookeeper.ZKUtil(164): regionserver:46629-0x10099497e1c0001, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-28 04:57:04,842 DEBUG [Listener at localhost/36897] zookeeper.ZKUtil(164): regionserver:46629-0x10099497e1c0001, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-28 04:57:04,843 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46629 2023-02-28 04:57:04,843 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46629 2023-02-28 04:57:04,843 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46629 2023-02-28 04:57:04,845 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46629 2023-02-28 04:57:04,845 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46629 2023-02-28 04:57:04,857 INFO [Listener at localhost/36897] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-28 04:57:04,857 INFO [Listener at localhost/36897] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-28 04:57:04,858 INFO [Listener at localhost/36897] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-28 04:57:04,858 INFO [Listener at localhost/36897] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-28 04:57:04,858 INFO [Listener at localhost/36897] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-28 04:57:04,858 INFO [Listener at localhost/36897] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-28 04:57:04,858 INFO [Listener at localhost/36897] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-28 04:57:04,860 INFO [Listener at localhost/36897] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36597 2023-02-28 04:57:04,860 INFO [Listener at localhost/36897] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-28 04:57:04,861 INFO [Listener at localhost/36897] mob.MobFileCache(124): MobFileCache disabled 2023-02-28 04:57:04,862 INFO [Listener at localhost/36897] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-28 04:57:04,864 INFO [Listener at localhost/36897] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-28 04:57:04,866 INFO [Listener at localhost/36897] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36597 connecting to ZooKeeper ensemble=127.0.0.1:54355 2023-02-28 04:57:04,870 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:365970x0, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-28 04:57:04,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:36597-0x10099497e1c0002 connected 2023-02-28 04:57:04,871 DEBUG [Listener at localhost/36897] zookeeper.ZKUtil(164): regionserver:36597-0x10099497e1c0002, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-28 04:57:04,872 DEBUG [Listener at localhost/36897] zookeeper.ZKUtil(164): regionserver:36597-0x10099497e1c0002, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-28 04:57:04,873 DEBUG [Listener at localhost/36897] zookeeper.ZKUtil(164): regionserver:36597-0x10099497e1c0002, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-28 04:57:04,873 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36597 2023-02-28 04:57:04,874 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36597 2023-02-28 04:57:04,874 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36597 2023-02-28 04:57:04,875 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36597 2023-02-28 04:57:04,875 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36597 2023-02-28 04:57:04,887 INFO [Listener at localhost/36897] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-28 04:57:04,887 INFO [Listener at localhost/36897] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-28 04:57:04,888 INFO [Listener at localhost/36897] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-28 04:57:04,888 INFO [Listener at localhost/36897] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-28 04:57:04,888 INFO [Listener at localhost/36897] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-28 04:57:04,888 INFO [Listener at localhost/36897] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-28 04:57:04,888 INFO [Listener at localhost/36897] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-28 04:57:04,889 INFO [Listener at localhost/36897] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38247 2023-02-28 04:57:04,890 INFO [Listener at localhost/36897] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-28 04:57:04,890 INFO [Listener at localhost/36897] mob.MobFileCache(124): MobFileCache disabled 2023-02-28 04:57:04,891 INFO [Listener at localhost/36897] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-28 04:57:04,893 INFO [Listener at localhost/36897] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-28 04:57:04,894 INFO [Listener at localhost/36897] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38247 connecting to ZooKeeper ensemble=127.0.0.1:54355 2023-02-28 04:57:04,897 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:382470x0, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-28 04:57:04,898 DEBUG [Listener at localhost/36897] zookeeper.ZKUtil(164): regionserver:382470x0, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-28 04:57:04,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:38247-0x10099497e1c0003 connected 2023-02-28 04:57:04,899 DEBUG [Listener at localhost/36897] zookeeper.ZKUtil(164): regionserver:38247-0x10099497e1c0003, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-28 04:57:04,899 DEBUG [Listener at localhost/36897] zookeeper.ZKUtil(164): regionserver:38247-0x10099497e1c0003, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-28 04:57:04,900 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38247 2023-02-28 04:57:04,900 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38247 2023-02-28 04:57:04,901 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38247 2023-02-28 04:57:04,901 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38247 2023-02-28 04:57:04,901 DEBUG [Listener at localhost/36897] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38247 2023-02-28 04:57:04,903 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,42505,1677560223662 2023-02-28 04:57:04,913 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-28 04:57:04,914 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,42505,1677560223662 2023-02-28 04:57:04,933 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-28 04:57:04,933 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:36597-0x10099497e1c0002, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-28 04:57:04,933 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:38247-0x10099497e1c0003, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-28 04:57:04,933 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:46629-0x10099497e1c0001, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-28 04:57:04,933 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-28 04:57:04,935 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-28 04:57:04,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-28 04:57:04,936 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,42505,1677560223662 from backup master directory 2023-02-28 04:57:04,940 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,42505,1677560223662 2023-02-28 04:57:04,940 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-28 04:57:04,940 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-28 04:57:04,941 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,42505,1677560223662 2023-02-28 04:57:04,943 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-28 04:57:04,945 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-28 04:57:05,035 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/hbase.id with ID: b6503725-5deb-4e94-acfb-d5582d72bb58 2023-02-28 04:57:05,077 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-28 04:57:05,091 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-28 04:57:05,138 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x4f55e535 to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:05,171 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1aee5f70, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:05,194 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-28 04:57:05,196 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-28 04:57:05,212 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-28 04:57:05,212 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-28 04:57:05,215 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-02-28 04:57:05,220 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-02-28 04:57:05,222 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-28 04:57:05,255 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/MasterData/data/master/store-tmp 2023-02-28 04:57:05,292 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:05,292 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-28 04:57:05,292 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-28 04:57:05,292 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-28 04:57:05,293 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-28 04:57:05,293 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-28 04:57:05,293 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-28 04:57:05,293 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-02-28 04:57:05,294 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/MasterData/WALs/jenkins-hbase4.apache.org,42505,1677560223662 2023-02-28 04:57:05,313 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42505%2C1677560223662, suffix=, logDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/MasterData/WALs/jenkins-hbase4.apache.org,42505,1677560223662, archiveDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/MasterData/oldWALs, maxLogs=10 2023-02-28 04:57:05,378 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37967,DS-399135da-b37d-4a80-ae3b-3bbfc6e63945,DISK] 2023-02-28 04:57:05,378 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37889,DS-994241a6-57fe-4042-bd49-62f8a7d0e6ea,DISK] 2023-02-28 04:57:05,378 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34137,DS-9e76f0e4-3b34-423f-b2ed-93f1bf763d0e,DISK] 2023-02-28 04:57:05,386 DEBUG [RS-EventLoopGroup-5-1] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-28 04:57:05,453 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/MasterData/WALs/jenkins-hbase4.apache.org,42505,1677560223662/jenkins-hbase4.apache.org%2C42505%2C1677560223662.1677560225323 2023-02-28 04:57:05,454 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34137,DS-9e76f0e4-3b34-423f-b2ed-93f1bf763d0e,DISK], DatanodeInfoWithStorage[127.0.0.1:37889,DS-994241a6-57fe-4042-bd49-62f8a7d0e6ea,DISK], DatanodeInfoWithStorage[127.0.0.1:37967,DS-399135da-b37d-4a80-ae3b-3bbfc6e63945,DISK]] 2023-02-28 04:57:05,454 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-28 04:57:05,454 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:05,457 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-28 04:57:05,458 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-28 04:57:05,513 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-28 04:57:05,520 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-28 04:57:05,544 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-28 04:57:05,556 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:05,563 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-28 04:57:05,565 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-28 04:57:05,579 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-28 04:57:05,583 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-28 04:57:05,584 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71427530, jitterRate=0.06435313820838928}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:05,584 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-02-28 04:57:05,585 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-28 04:57:05,604 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-28 04:57:05,604 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-28 04:57:05,606 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-28 04:57:05,608 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-02-28 04:57:05,639 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-02-28 04:57:05,639 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-28 04:57:05,665 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-28 04:57:05,670 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-28 04:57:05,693 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-28 04:57:05,696 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-28 04:57:05,698 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-28 04:57:05,703 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-28 04:57:05,707 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-28 04:57:05,709 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-28 04:57:05,711 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-28 04:57:05,711 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-28 04:57:05,723 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-28 04:57:05,727 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-28 04:57:05,727 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:46629-0x10099497e1c0001, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-28 04:57:05,727 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:38247-0x10099497e1c0003, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-28 04:57:05,727 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:36597-0x10099497e1c0002, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-28 04:57:05,728 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-28 04:57:05,728 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,42505,1677560223662, sessionid=0x10099497e1c0000, setting cluster-up flag (Was=false) 2023-02-28 04:57:05,744 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-28 04:57:05,750 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-28 04:57:05,751 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42505,1677560223662 2023-02-28 04:57:05,758 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-28 04:57:05,764 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-28 04:57:05,765 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,42505,1677560223662 2023-02-28 04:57:05,767 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp 2023-02-28 04:57:05,806 INFO [RS:1;jenkins-hbase4:36597] regionserver.HRegionServer(952): ClusterId : b6503725-5deb-4e94-acfb-d5582d72bb58 2023-02-28 04:57:05,806 INFO [RS:0;jenkins-hbase4:46629] regionserver.HRegionServer(952): ClusterId : b6503725-5deb-4e94-acfb-d5582d72bb58 2023-02-28 04:57:05,806 INFO [RS:2;jenkins-hbase4:38247] regionserver.HRegionServer(952): ClusterId : b6503725-5deb-4e94-acfb-d5582d72bb58 2023-02-28 04:57:05,813 DEBUG [RS:1;jenkins-hbase4:36597] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-28 04:57:05,813 DEBUG [RS:2;jenkins-hbase4:38247] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-28 04:57:05,813 DEBUG [RS:0;jenkins-hbase4:46629] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-28 04:57:05,820 DEBUG [RS:2;jenkins-hbase4:38247] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-28 04:57:05,820 DEBUG [RS:1;jenkins-hbase4:36597] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-28 04:57:05,820 DEBUG [RS:0;jenkins-hbase4:46629] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-28 04:57:05,820 DEBUG [RS:1;jenkins-hbase4:36597] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-28 04:57:05,820 DEBUG [RS:2;jenkins-hbase4:38247] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-28 04:57:05,820 DEBUG [RS:0;jenkins-hbase4:46629] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-28 04:57:05,825 DEBUG [RS:0;jenkins-hbase4:46629] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-28 04:57:05,825 DEBUG [RS:2;jenkins-hbase4:38247] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-28 04:57:05,825 DEBUG [RS:1;jenkins-hbase4:36597] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-28 04:57:05,827 DEBUG [RS:2;jenkins-hbase4:38247] zookeeper.ReadOnlyZKClient(139): Connect 0x1e3b6661 to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:05,827 DEBUG [RS:0;jenkins-hbase4:46629] zookeeper.ReadOnlyZKClient(139): Connect 0x365445cb to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:05,827 DEBUG [RS:1;jenkins-hbase4:36597] zookeeper.ReadOnlyZKClient(139): Connect 0x76b7cd6b to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:05,853 DEBUG [RS:1;jenkins-hbase4:36597] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78702dcd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:05,854 DEBUG [RS:2;jenkins-hbase4:38247] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10fc9bd6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:05,854 DEBUG [RS:1;jenkins-hbase4:36597] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c4fa61b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-28 04:57:05,855 DEBUG [RS:2;jenkins-hbase4:38247] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14c30d11, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-28 04:57:05,855 DEBUG [RS:0;jenkins-hbase4:46629] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3893bd71, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:05,855 DEBUG [RS:0;jenkins-hbase4:46629] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cac2ebb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-28 04:57:05,878 DEBUG [RS:1;jenkins-hbase4:36597] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:36597 2023-02-28 04:57:05,881 INFO [RS:1;jenkins-hbase4:36597] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-28 04:57:05,881 INFO [RS:1;jenkins-hbase4:36597] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-28 04:57:05,881 DEBUG [RS:1;jenkins-hbase4:36597] regionserver.HRegionServer(1023): About to register with Master. 2023-02-28 04:57:05,882 DEBUG [RS:0;jenkins-hbase4:46629] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:46629 2023-02-28 04:57:05,882 DEBUG [RS:2;jenkins-hbase4:38247] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:38247 2023-02-28 04:57:05,883 INFO [RS:0;jenkins-hbase4:46629] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-28 04:57:05,883 INFO [RS:2;jenkins-hbase4:38247] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-28 04:57:05,883 INFO [RS:2;jenkins-hbase4:38247] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-28 04:57:05,883 DEBUG [RS:2;jenkins-hbase4:38247] regionserver.HRegionServer(1023): About to register with Master. 2023-02-28 04:57:05,883 INFO [RS:0;jenkins-hbase4:46629] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-28 04:57:05,884 DEBUG [RS:0;jenkins-hbase4:46629] regionserver.HRegionServer(1023): About to register with Master. 2023-02-28 04:57:05,886 INFO [RS:1;jenkins-hbase4:36597] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,42505,1677560223662 with isa=jenkins-hbase4.apache.org/172.31.14.131:36597, startcode=1677560224857 2023-02-28 04:57:05,886 INFO [RS:0;jenkins-hbase4:46629] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,42505,1677560223662 with isa=jenkins-hbase4.apache.org/172.31.14.131:46629, startcode=1677560224813 2023-02-28 04:57:05,886 INFO [RS:2;jenkins-hbase4:38247] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,42505,1677560223662 with isa=jenkins-hbase4.apache.org/172.31.14.131:38247, startcode=1677560224887 2023-02-28 04:57:05,892 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-28 04:57:05,903 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-28 04:57:05,903 DEBUG [RS:0;jenkins-hbase4:46629] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-28 04:57:05,903 DEBUG [RS:2;jenkins-hbase4:38247] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-28 04:57:05,903 DEBUG [RS:1;jenkins-hbase4:36597] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-28 04:57:05,903 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-28 04:57:05,905 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-28 04:57:05,905 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-28 04:57:05,905 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-28 04:57:05,905 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:05,905 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-28 04:57:05,905 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:05,906 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1677560255906 2023-02-28 04:57:05,908 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-28 04:57:05,913 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-28 04:57:05,914 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-28 04:57:05,919 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-28 04:57:05,922 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-28 04:57:05,929 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-28 04:57:05,930 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-28 04:57:05,930 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-28 04:57:05,930 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-28 04:57:05,931 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:05,933 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-28 04:57:05,934 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-28 04:57:05,935 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-28 04:57:05,937 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-28 04:57:05,938 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-28 04:57:05,940 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1677560225940,5,FailOnTimeoutGroup] 2023-02-28 04:57:05,944 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1677560225940,5,FailOnTimeoutGroup] 2023-02-28 04:57:05,944 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:05,944 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-28 04:57:05,945 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:05,946 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:05,975 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-28 04:57:05,976 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-28 04:57:05,977 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f 2023-02-28 04:57:05,977 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37581, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-28 04:57:05,977 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37583, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-28 04:57:05,977 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51163, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-28 04:57:05,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42505] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:05,993 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:05,994 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42505] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:06,007 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:06,010 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-28 04:57:06,014 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/info 2023-02-28 04:57:06,015 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-28 04:57:06,015 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:06,016 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-28 04:57:06,017 DEBUG [RS:1;jenkins-hbase4:36597] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f 2023-02-28 04:57:06,017 DEBUG [RS:0;jenkins-hbase4:46629] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f 2023-02-28 04:57:06,017 DEBUG [RS:2;jenkins-hbase4:38247] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f 2023-02-28 04:57:06,018 DEBUG [RS:0;jenkins-hbase4:46629] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:33733 2023-02-28 04:57:06,018 DEBUG [RS:1;jenkins-hbase4:36597] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:33733 2023-02-28 04:57:06,018 DEBUG [RS:0;jenkins-hbase4:46629] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-28 04:57:06,018 DEBUG [RS:2;jenkins-hbase4:38247] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:33733 2023-02-28 04:57:06,018 DEBUG [RS:1;jenkins-hbase4:36597] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-28 04:57:06,018 DEBUG [RS:2;jenkins-hbase4:38247] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-28 04:57:06,019 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/rep_barrier 2023-02-28 04:57:06,020 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-28 04:57:06,021 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:06,021 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-28 04:57:06,024 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/table 2023-02-28 04:57:06,024 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-28 04:57:06,025 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:06,027 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740 2023-02-28 04:57:06,027 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740 2023-02-28 04:57:06,031 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-28 04:57:06,031 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-28 04:57:06,032 DEBUG [RS:0;jenkins-hbase4:46629] zookeeper.ZKUtil(162): regionserver:46629-0x10099497e1c0001, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:06,032 DEBUG [RS:2;jenkins-hbase4:38247] zookeeper.ZKUtil(162): regionserver:38247-0x10099497e1c0003, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:06,032 DEBUG [RS:1;jenkins-hbase4:36597] zookeeper.ZKUtil(162): regionserver:36597-0x10099497e1c0002, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:06,032 WARN [RS:2;jenkins-hbase4:38247] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-28 04:57:06,032 WARN [RS:0;jenkins-hbase4:46629] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-28 04:57:06,033 INFO [RS:2;jenkins-hbase4:38247] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-28 04:57:06,033 INFO [RS:0;jenkins-hbase4:46629] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-28 04:57:06,033 WARN [RS:1;jenkins-hbase4:36597] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-28 04:57:06,033 DEBUG [RS:0;jenkins-hbase4:46629] regionserver.HRegionServer(1947): logDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/WALs/jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:06,033 INFO [RS:1;jenkins-hbase4:36597] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-28 04:57:06,033 DEBUG [RS:2;jenkins-hbase4:38247] regionserver.HRegionServer(1947): logDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/WALs/jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:06,034 DEBUG [RS:1;jenkins-hbase4:36597] regionserver.HRegionServer(1947): logDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/WALs/jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:06,034 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,38247,1677560224887] 2023-02-28 04:57:06,034 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,36597,1677560224857] 2023-02-28 04:57:06,034 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,46629,1677560224813] 2023-02-28 04:57:06,035 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-28 04:57:06,049 DEBUG [RS:2;jenkins-hbase4:38247] zookeeper.ZKUtil(162): regionserver:38247-0x10099497e1c0003, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:06,049 DEBUG [RS:1;jenkins-hbase4:36597] zookeeper.ZKUtil(162): regionserver:36597-0x10099497e1c0002, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:06,050 DEBUG [RS:0;jenkins-hbase4:46629] zookeeper.ZKUtil(162): regionserver:46629-0x10099497e1c0001, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:06,050 DEBUG [RS:2;jenkins-hbase4:38247] zookeeper.ZKUtil(162): regionserver:38247-0x10099497e1c0003, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:06,050 DEBUG [RS:1;jenkins-hbase4:36597] zookeeper.ZKUtil(162): regionserver:36597-0x10099497e1c0002, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:06,050 DEBUG [RS:0;jenkins-hbase4:46629] zookeeper.ZKUtil(162): regionserver:46629-0x10099497e1c0001, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:06,051 DEBUG [RS:2;jenkins-hbase4:38247] zookeeper.ZKUtil(162): regionserver:38247-0x10099497e1c0003, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:06,051 DEBUG [RS:1;jenkins-hbase4:36597] zookeeper.ZKUtil(162): regionserver:36597-0x10099497e1c0002, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:06,052 DEBUG [RS:0;jenkins-hbase4:46629] zookeeper.ZKUtil(162): regionserver:46629-0x10099497e1c0001, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:06,053 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-28 04:57:06,054 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75472095, jitterRate=0.12462185323238373}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-28 04:57:06,054 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: 2023-02-28 04:57:06,054 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-28 04:57:06,054 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-28 04:57:06,054 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-28 04:57:06,054 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-28 04:57:06,054 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-28 04:57:06,056 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-28 04:57:06,056 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: 2023-02-28 04:57:06,061 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-28 04:57:06,062 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-28 04:57:06,065 DEBUG [RS:1;jenkins-hbase4:36597] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-28 04:57:06,065 DEBUG [RS:2;jenkins-hbase4:38247] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-28 04:57:06,065 DEBUG [RS:0;jenkins-hbase4:46629] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-28 04:57:06,072 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-28 04:57:06,076 INFO [RS:0;jenkins-hbase4:46629] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-28 04:57:06,076 INFO [RS:1;jenkins-hbase4:36597] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-28 04:57:06,077 INFO [RS:2;jenkins-hbase4:38247] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-28 04:57:06,086 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-28 04:57:06,090 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:06,102 INFO [RS:1;jenkins-hbase4:36597] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-28 04:57:06,102 INFO [RS:0;jenkins-hbase4:46629] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-28 04:57:06,102 INFO [RS:2;jenkins-hbase4:38247] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-28 04:57:06,106 INFO [RS:0;jenkins-hbase4:46629] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-28 04:57:06,106 INFO [RS:2;jenkins-hbase4:38247] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-28 04:57:06,106 INFO [RS:0;jenkins-hbase4:46629] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,106 INFO [RS:1;jenkins-hbase4:36597] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-28 04:57:06,106 INFO [RS:2;jenkins-hbase4:38247] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,107 INFO [RS:1;jenkins-hbase4:36597] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,107 INFO [RS:0;jenkins-hbase4:46629] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-28 04:57:06,107 INFO [RS:2;jenkins-hbase4:38247] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-28 04:57:06,107 INFO [RS:1;jenkins-hbase4:36597] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-28 04:57:06,116 INFO [RS:2;jenkins-hbase4:38247] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,116 INFO [RS:1;jenkins-hbase4:36597] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,116 INFO [RS:0;jenkins-hbase4:46629] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,116 DEBUG [RS:2;jenkins-hbase4:38247] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,116 DEBUG [RS:1;jenkins-hbase4:36597] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,116 DEBUG [RS:2;jenkins-hbase4:38247] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,117 DEBUG [RS:1;jenkins-hbase4:36597] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,117 DEBUG [RS:2;jenkins-hbase4:38247] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,117 DEBUG [RS:1;jenkins-hbase4:36597] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,117 DEBUG [RS:2;jenkins-hbase4:38247] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,117 DEBUG [RS:1;jenkins-hbase4:36597] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,117 DEBUG [RS:2;jenkins-hbase4:38247] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,117 DEBUG [RS:1;jenkins-hbase4:36597] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,117 DEBUG [RS:2;jenkins-hbase4:38247] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-28 04:57:06,117 DEBUG [RS:1;jenkins-hbase4:36597] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-28 04:57:06,117 DEBUG [RS:2;jenkins-hbase4:38247] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,117 DEBUG [RS:1;jenkins-hbase4:36597] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,116 DEBUG [RS:0;jenkins-hbase4:46629] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,117 DEBUG [RS:1;jenkins-hbase4:36597] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,117 DEBUG [RS:0;jenkins-hbase4:46629] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,117 DEBUG [RS:1;jenkins-hbase4:36597] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,117 DEBUG [RS:0;jenkins-hbase4:46629] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,117 DEBUG [RS:1;jenkins-hbase4:36597] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,117 DEBUG [RS:2;jenkins-hbase4:38247] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,117 DEBUG [RS:0;jenkins-hbase4:46629] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,118 DEBUG [RS:2;jenkins-hbase4:38247] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,118 DEBUG [RS:0;jenkins-hbase4:46629] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,118 DEBUG [RS:2;jenkins-hbase4:38247] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,118 DEBUG [RS:0;jenkins-hbase4:46629] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-28 04:57:06,118 DEBUG [RS:0;jenkins-hbase4:46629] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,118 DEBUG [RS:0;jenkins-hbase4:46629] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,118 DEBUG [RS:0;jenkins-hbase4:46629] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,118 DEBUG [RS:0;jenkins-hbase4:46629] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-28 04:57:06,119 INFO [RS:0;jenkins-hbase4:46629] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,119 INFO [RS:2;jenkins-hbase4:38247] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,119 INFO [RS:1;jenkins-hbase4:36597] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,120 INFO [RS:2;jenkins-hbase4:38247] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,120 INFO [RS:1;jenkins-hbase4:36597] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,119 INFO [RS:0;jenkins-hbase4:46629] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,120 INFO [RS:1;jenkins-hbase4:36597] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,120 INFO [RS:0;jenkins-hbase4:46629] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,120 INFO [RS:2;jenkins-hbase4:38247] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,138 INFO [RS:0;jenkins-hbase4:46629] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-28 04:57:06,141 INFO [RS:2;jenkins-hbase4:38247] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-28 04:57:06,141 INFO [RS:1;jenkins-hbase4:36597] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-28 04:57:06,141 INFO [RS:1;jenkins-hbase4:36597] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36597,1677560224857-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,141 INFO [RS:0;jenkins-hbase4:46629] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46629,1677560224813-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,141 INFO [RS:2;jenkins-hbase4:38247] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38247,1677560224887-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,157 INFO [RS:0;jenkins-hbase4:46629] regionserver.Replication(203): jenkins-hbase4.apache.org,46629,1677560224813 started 2023-02-28 04:57:06,157 INFO [RS:0;jenkins-hbase4:46629] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,46629,1677560224813, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:46629, sessionid=0x10099497e1c0001 2023-02-28 04:57:06,157 DEBUG [RS:0;jenkins-hbase4:46629] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-28 04:57:06,157 DEBUG [RS:0;jenkins-hbase4:46629] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:06,158 DEBUG [RS:0;jenkins-hbase4:46629] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46629,1677560224813' 2023-02-28 04:57:06,158 DEBUG [RS:0;jenkins-hbase4:46629] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-28 04:57:06,158 DEBUG [RS:0;jenkins-hbase4:46629] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-28 04:57:06,159 DEBUG [RS:0;jenkins-hbase4:46629] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-28 04:57:06,159 DEBUG [RS:0;jenkins-hbase4:46629] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-28 04:57:06,159 DEBUG [RS:0;jenkins-hbase4:46629] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:06,159 DEBUG [RS:0;jenkins-hbase4:46629] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,46629,1677560224813' 2023-02-28 04:57:06,159 DEBUG [RS:0;jenkins-hbase4:46629] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-28 04:57:06,159 DEBUG [RS:0;jenkins-hbase4:46629] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-28 04:57:06,160 DEBUG [RS:0;jenkins-hbase4:46629] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-28 04:57:06,160 INFO [RS:0;jenkins-hbase4:46629] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-28 04:57:06,160 INFO [RS:0;jenkins-hbase4:46629] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-28 04:57:06,161 INFO [RS:1;jenkins-hbase4:36597] regionserver.Replication(203): jenkins-hbase4.apache.org,36597,1677560224857 started 2023-02-28 04:57:06,161 INFO [RS:2;jenkins-hbase4:38247] regionserver.Replication(203): jenkins-hbase4.apache.org,38247,1677560224887 started 2023-02-28 04:57:06,161 INFO [RS:1;jenkins-hbase4:36597] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,36597,1677560224857, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:36597, sessionid=0x10099497e1c0002 2023-02-28 04:57:06,161 INFO [RS:2;jenkins-hbase4:38247] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,38247,1677560224887, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:38247, sessionid=0x10099497e1c0003 2023-02-28 04:57:06,161 DEBUG [RS:1;jenkins-hbase4:36597] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-28 04:57:06,161 DEBUG [RS:2;jenkins-hbase4:38247] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-28 04:57:06,162 DEBUG [RS:2;jenkins-hbase4:38247] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:06,161 DEBUG [RS:1;jenkins-hbase4:36597] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:06,163 DEBUG [RS:1;jenkins-hbase4:36597] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36597,1677560224857' 2023-02-28 04:57:06,162 DEBUG [RS:2;jenkins-hbase4:38247] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38247,1677560224887' 2023-02-28 04:57:06,163 DEBUG [RS:2;jenkins-hbase4:38247] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-28 04:57:06,163 DEBUG [RS:1;jenkins-hbase4:36597] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-28 04:57:06,163 DEBUG [RS:2;jenkins-hbase4:38247] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-28 04:57:06,163 DEBUG [RS:1;jenkins-hbase4:36597] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-28 04:57:06,164 DEBUG [RS:2;jenkins-hbase4:38247] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-28 04:57:06,164 DEBUG [RS:1;jenkins-hbase4:36597] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-28 04:57:06,164 DEBUG [RS:2;jenkins-hbase4:38247] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-28 04:57:06,164 DEBUG [RS:1;jenkins-hbase4:36597] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-28 04:57:06,164 DEBUG [RS:1;jenkins-hbase4:36597] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:06,164 DEBUG [RS:2;jenkins-hbase4:38247] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:06,164 DEBUG [RS:2;jenkins-hbase4:38247] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38247,1677560224887' 2023-02-28 04:57:06,164 DEBUG [RS:1;jenkins-hbase4:36597] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36597,1677560224857' 2023-02-28 04:57:06,165 DEBUG [RS:1;jenkins-hbase4:36597] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-28 04:57:06,165 DEBUG [RS:2;jenkins-hbase4:38247] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-28 04:57:06,165 DEBUG [RS:1;jenkins-hbase4:36597] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-28 04:57:06,165 DEBUG [RS:2;jenkins-hbase4:38247] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-28 04:57:06,166 DEBUG [RS:1;jenkins-hbase4:36597] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-28 04:57:06,166 DEBUG [RS:2;jenkins-hbase4:38247] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-28 04:57:06,166 INFO [RS:1;jenkins-hbase4:36597] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-28 04:57:06,167 INFO [RS:2;jenkins-hbase4:38247] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-28 04:57:06,167 INFO [RS:2;jenkins-hbase4:38247] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-28 04:57:06,167 INFO [RS:1;jenkins-hbase4:36597] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-28 04:57:06,241 DEBUG [jenkins-hbase4:42505] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-28 04:57:06,245 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-28 04:57:06,251 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-28 04:57:06,251 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-28 04:57:06,251 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-28 04:57:06,251 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-28 04:57:06,254 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,38247,1677560224887, state=OPENING 2023-02-28 04:57:06,261 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-28 04:57:06,262 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-28 04:57:06,263 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-28 04:57:06,267 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:06,272 INFO [RS:1;jenkins-hbase4:36597] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36597%2C1677560224857, suffix=, logDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/WALs/jenkins-hbase4.apache.org,36597,1677560224857, archiveDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/oldWALs, maxLogs=32 2023-02-28 04:57:06,272 INFO [RS:2;jenkins-hbase4:38247] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38247%2C1677560224887, suffix=, logDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/WALs/jenkins-hbase4.apache.org,38247,1677560224887, archiveDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/oldWALs, maxLogs=32 2023-02-28 04:57:06,272 INFO [RS:0;jenkins-hbase4:46629] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46629%2C1677560224813, suffix=, logDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/WALs/jenkins-hbase4.apache.org,46629,1677560224813, archiveDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/oldWALs, maxLogs=32 2023-02-28 04:57:06,298 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37889,DS-994241a6-57fe-4042-bd49-62f8a7d0e6ea,DISK] 2023-02-28 04:57:06,298 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34137,DS-9e76f0e4-3b34-423f-b2ed-93f1bf763d0e,DISK] 2023-02-28 04:57:06,298 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37967,DS-399135da-b37d-4a80-ae3b-3bbfc6e63945,DISK] 2023-02-28 04:57:06,306 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37967,DS-399135da-b37d-4a80-ae3b-3bbfc6e63945,DISK] 2023-02-28 04:57:06,306 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37889,DS-994241a6-57fe-4042-bd49-62f8a7d0e6ea,DISK] 2023-02-28 04:57:06,306 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34137,DS-9e76f0e4-3b34-423f-b2ed-93f1bf763d0e,DISK] 2023-02-28 04:57:06,314 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34137,DS-9e76f0e4-3b34-423f-b2ed-93f1bf763d0e,DISK] 2023-02-28 04:57:06,314 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37889,DS-994241a6-57fe-4042-bd49-62f8a7d0e6ea,DISK] 2023-02-28 04:57:06,315 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37967,DS-399135da-b37d-4a80-ae3b-3bbfc6e63945,DISK] 2023-02-28 04:57:06,316 INFO [RS:1;jenkins-hbase4:36597] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/WALs/jenkins-hbase4.apache.org,36597,1677560224857/jenkins-hbase4.apache.org%2C36597%2C1677560224857.1677560226277 2023-02-28 04:57:06,319 DEBUG [RS:1;jenkins-hbase4:36597] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37889,DS-994241a6-57fe-4042-bd49-62f8a7d0e6ea,DISK], DatanodeInfoWithStorage[127.0.0.1:37967,DS-399135da-b37d-4a80-ae3b-3bbfc6e63945,DISK], DatanodeInfoWithStorage[127.0.0.1:34137,DS-9e76f0e4-3b34-423f-b2ed-93f1bf763d0e,DISK]] 2023-02-28 04:57:06,321 INFO [RS:2;jenkins-hbase4:38247] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/WALs/jenkins-hbase4.apache.org,38247,1677560224887/jenkins-hbase4.apache.org%2C38247%2C1677560224887.1677560226278 2023-02-28 04:57:06,322 DEBUG [RS:2;jenkins-hbase4:38247] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37889,DS-994241a6-57fe-4042-bd49-62f8a7d0e6ea,DISK], DatanodeInfoWithStorage[127.0.0.1:34137,DS-9e76f0e4-3b34-423f-b2ed-93f1bf763d0e,DISK], DatanodeInfoWithStorage[127.0.0.1:37967,DS-399135da-b37d-4a80-ae3b-3bbfc6e63945,DISK]] 2023-02-28 04:57:06,322 INFO [RS:0;jenkins-hbase4:46629] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/WALs/jenkins-hbase4.apache.org,46629,1677560224813/jenkins-hbase4.apache.org%2C46629%2C1677560224813.1677560226277 2023-02-28 04:57:06,322 DEBUG [RS:0;jenkins-hbase4:46629] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37889,DS-994241a6-57fe-4042-bd49-62f8a7d0e6ea,DISK], DatanodeInfoWithStorage[127.0.0.1:34137,DS-9e76f0e4-3b34-423f-b2ed-93f1bf763d0e,DISK], DatanodeInfoWithStorage[127.0.0.1:37967,DS-399135da-b37d-4a80-ae3b-3bbfc6e63945,DISK]] 2023-02-28 04:57:06,458 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:06,460 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-28 04:57:06,463 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49956, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-28 04:57:06,475 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-28 04:57:06,476 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-28 04:57:06,479 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38247%2C1677560224887.meta, suffix=.meta, logDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/WALs/jenkins-hbase4.apache.org,38247,1677560224887, archiveDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/oldWALs, maxLogs=32 2023-02-28 04:57:06,504 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37967,DS-399135da-b37d-4a80-ae3b-3bbfc6e63945,DISK] 2023-02-28 04:57:06,504 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34137,DS-9e76f0e4-3b34-423f-b2ed-93f1bf763d0e,DISK] 2023-02-28 04:57:06,506 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37889,DS-994241a6-57fe-4042-bd49-62f8a7d0e6ea,DISK] 2023-02-28 04:57:06,512 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/WALs/jenkins-hbase4.apache.org,38247,1677560224887/jenkins-hbase4.apache.org%2C38247%2C1677560224887.meta.1677560226482.meta 2023-02-28 04:57:06,512 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37967,DS-399135da-b37d-4a80-ae3b-3bbfc6e63945,DISK], DatanodeInfoWithStorage[127.0.0.1:34137,DS-9e76f0e4-3b34-423f-b2ed-93f1bf763d0e,DISK], DatanodeInfoWithStorage[127.0.0.1:37889,DS-994241a6-57fe-4042-bd49-62f8a7d0e6ea,DISK]] 2023-02-28 04:57:06,513 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-28 04:57:06,515 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-28 04:57:06,535 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-28 04:57:06,540 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-28 04:57:06,547 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-28 04:57:06,548 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:06,548 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-28 04:57:06,548 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-28 04:57:06,552 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-28 04:57:06,554 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/info 2023-02-28 04:57:06,554 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/info 2023-02-28 04:57:06,554 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-28 04:57:06,555 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:06,555 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-28 04:57:06,557 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/rep_barrier 2023-02-28 04:57:06,557 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/rep_barrier 2023-02-28 04:57:06,557 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-28 04:57:06,558 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:06,558 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-28 04:57:06,560 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/table 2023-02-28 04:57:06,560 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/table 2023-02-28 04:57:06,560 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-28 04:57:06,561 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:06,563 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740 2023-02-28 04:57:06,565 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740 2023-02-28 04:57:06,568 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-28 04:57:06,571 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-28 04:57:06,572 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69923350, jitterRate=0.041939109563827515}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-28 04:57:06,572 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: 2023-02-28 04:57:06,582 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1677560226450 2023-02-28 04:57:06,599 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-28 04:57:06,600 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-28 04:57:06,601 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,38247,1677560224887, state=OPEN 2023-02-28 04:57:06,604 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-28 04:57:06,604 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-28 04:57:06,608 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-28 04:57:06,608 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,38247,1677560224887 in 337 msec 2023-02-28 04:57:06,613 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-28 04:57:06,613 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 537 msec 2023-02-28 04:57:06,618 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 805 msec 2023-02-28 04:57:06,618 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1677560226618, completionTime=-1 2023-02-28 04:57:06,618 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-28 04:57:06,618 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-28 04:57:06,688 DEBUG [hconnection-0x5d8659e3-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:06,691 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49968, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:06,709 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-28 04:57:06,709 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1677560286709 2023-02-28 04:57:06,709 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1677560346709 2023-02-28 04:57:06,709 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 91 msec 2023-02-28 04:57:06,734 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42505,1677560223662-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,735 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42505,1677560223662-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,735 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42505,1677560223662-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,737 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:42505, period=300000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,738 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-28 04:57:06,746 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-28 04:57:06,758 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-28 04:57:06,760 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-28 04:57:06,770 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-28 04:57:06,773 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-28 04:57:06,775 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-28 04:57:06,797 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/hbase/namespace/befed80c35fc04cd1003e1970c424534 2023-02-28 04:57:06,799 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/hbase/namespace/befed80c35fc04cd1003e1970c424534 empty. 2023-02-28 04:57:06,800 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/hbase/namespace/befed80c35fc04cd1003e1970c424534 2023-02-28 04:57:06,800 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-28 04:57:06,843 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-28 04:57:06,846 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => befed80c35fc04cd1003e1970c424534, NAME => 'hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:06,867 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:06,867 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing befed80c35fc04cd1003e1970c424534, disabling compactions & flushes 2023-02-28 04:57:06,867 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534. 2023-02-28 04:57:06,867 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534. 2023-02-28 04:57:06,867 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534. after waiting 0 ms 2023-02-28 04:57:06,867 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534. 2023-02-28 04:57:06,867 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534. 2023-02-28 04:57:06,868 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for befed80c35fc04cd1003e1970c424534: 2023-02-28 04:57:06,872 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-28 04:57:06,888 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1677560226875"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560226875"}]},"ts":"1677560226875"} 2023-02-28 04:57:06,915 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-28 04:57:06,917 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-28 04:57:06,921 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560226917"}]},"ts":"1677560226917"} 2023-02-28 04:57:06,926 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-28 04:57:06,930 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-28 04:57:06,932 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-28 04:57:06,932 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-28 04:57:06,932 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-28 04:57:06,932 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-28 04:57:06,934 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=befed80c35fc04cd1003e1970c424534, ASSIGN}] 2023-02-28 04:57:06,937 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=befed80c35fc04cd1003e1970c424534, ASSIGN 2023-02-28 04:57:06,938 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=befed80c35fc04cd1003e1970c424534, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38247,1677560224887; forceNewPlan=false, retain=false 2023-02-28 04:57:07,091 INFO [jenkins-hbase4:42505] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-28 04:57:07,092 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=befed80c35fc04cd1003e1970c424534, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:07,092 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1677560227091"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560227091"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560227091"}]},"ts":"1677560227091"} 2023-02-28 04:57:07,096 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure befed80c35fc04cd1003e1970c424534, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:07,259 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534. 2023-02-28 04:57:07,259 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => befed80c35fc04cd1003e1970c424534, NAME => 'hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534.', STARTKEY => '', ENDKEY => ''} 2023-02-28 04:57:07,260 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace befed80c35fc04cd1003e1970c424534 2023-02-28 04:57:07,260 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:07,260 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for befed80c35fc04cd1003e1970c424534 2023-02-28 04:57:07,260 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for befed80c35fc04cd1003e1970c424534 2023-02-28 04:57:07,262 INFO [StoreOpener-befed80c35fc04cd1003e1970c424534-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region befed80c35fc04cd1003e1970c424534 2023-02-28 04:57:07,265 DEBUG [StoreOpener-befed80c35fc04cd1003e1970c424534-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/namespace/befed80c35fc04cd1003e1970c424534/info 2023-02-28 04:57:07,265 DEBUG [StoreOpener-befed80c35fc04cd1003e1970c424534-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/namespace/befed80c35fc04cd1003e1970c424534/info 2023-02-28 04:57:07,265 INFO [StoreOpener-befed80c35fc04cd1003e1970c424534-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region befed80c35fc04cd1003e1970c424534 columnFamilyName info 2023-02-28 04:57:07,266 INFO [StoreOpener-befed80c35fc04cd1003e1970c424534-1] regionserver.HStore(310): Store=befed80c35fc04cd1003e1970c424534/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:07,268 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/namespace/befed80c35fc04cd1003e1970c424534 2023-02-28 04:57:07,269 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/namespace/befed80c35fc04cd1003e1970c424534 2023-02-28 04:57:07,273 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for befed80c35fc04cd1003e1970c424534 2023-02-28 04:57:07,276 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/namespace/befed80c35fc04cd1003e1970c424534/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-28 04:57:07,277 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened befed80c35fc04cd1003e1970c424534; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70255319, jitterRate=0.04688583314418793}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:07,277 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for befed80c35fc04cd1003e1970c424534: 2023-02-28 04:57:07,279 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534., pid=6, masterSystemTime=1677560227249 2023-02-28 04:57:07,282 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534. 2023-02-28 04:57:07,282 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534. 2023-02-28 04:57:07,284 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=befed80c35fc04cd1003e1970c424534, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:07,284 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1677560227283"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560227283"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560227283"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560227283"}]},"ts":"1677560227283"} 2023-02-28 04:57:07,291 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-28 04:57:07,291 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure befed80c35fc04cd1003e1970c424534, server=jenkins-hbase4.apache.org,38247,1677560224887 in 191 msec 2023-02-28 04:57:07,295 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-28 04:57:07,295 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=befed80c35fc04cd1003e1970c424534, ASSIGN in 357 msec 2023-02-28 04:57:07,297 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-28 04:57:07,297 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560227297"}]},"ts":"1677560227297"} 2023-02-28 04:57:07,299 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-28 04:57:07,304 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-28 04:57:07,307 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 543 msec 2023-02-28 04:57:07,373 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-28 04:57:07,375 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-28 04:57:07,375 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-28 04:57:07,408 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-28 04:57:07,424 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-28 04:57:07,431 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 30 msec 2023-02-28 04:57:07,441 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-28 04:57:07,453 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-28 04:57:07,459 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-02-28 04:57:07,467 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-28 04:57:07,470 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-28 04:57:07,472 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.531sec 2023-02-28 04:57:07,475 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-28 04:57:07,477 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-28 04:57:07,477 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-28 04:57:07,478 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42505,1677560223662-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-28 04:57:07,480 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42505,1677560223662-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-28 04:57:07,488 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-02-28 04:57:07,512 DEBUG [Listener at localhost/36897] zookeeper.ReadOnlyZKClient(139): Connect 0x0da5f7e5 to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:07,517 DEBUG [Listener at localhost/36897] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@492b920e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:07,530 DEBUG [hconnection-0x56d8d559-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:07,543 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49976, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:07,553 INFO [Listener at localhost/36897] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,42505,1677560223662 2023-02-28 04:57:07,582 INFO [Listener at localhost/36897] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=361, OpenFileDescriptor=688, MaxFileDescriptor=60000, SystemLoadAverage=227, ProcessCount=173, AvailableMemoryMB=9332 2023-02-28 04:57:07,596 DEBUG [Listener at localhost/36897] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-28 04:57:07,599 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33426, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-28 04:57:07,611 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'testRestoreSnapshot-1677560227585', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-28 04:57:07,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:07,615 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1677560227585 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-28 04:57:07,618 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot-1677560227585 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-28 04:57:07,620 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot-1677560227585" procId is: 9 2023-02-28 04:57:07,627 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:07,627 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:07,627 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:07,627 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:07,627 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:07,628 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:07,628 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 empty. 2023-02-28 04:57:07,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-28 04:57:07,628 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee empty. 2023-02-28 04:57:07,628 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 empty. 2023-02-28 04:57:07,629 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc empty. 2023-02-28 04:57:07,629 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 empty. 2023-02-28 04:57:07,631 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:07,631 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:07,632 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:07,632 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:07,629 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 empty. 2023-02-28 04:57:07,632 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:07,633 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:07,636 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1677560227585 regions 2023-02-28 04:57:07,658 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/.tabledesc/.tableinfo.0000000001 2023-02-28 04:57:07,660 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(7671): creating {ENCODED => bd66d5b12692aefad5974a6eb15a4f47, NAME => 'testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1677560227585', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:07,661 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(7671): creating {ENCODED => f60583096029c9557b712df1333409e7, NAME => 'testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1677560227585', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:07,662 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(7671): creating {ENCODED => 554a5845ea85f8afefc80a4e37e38fdc, NAME => 'testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1677560227585', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:07,741 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:07,742 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(1603): Closing bd66d5b12692aefad5974a6eb15a4f47, disabling compactions & flushes 2023-02-28 04:57:07,742 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:07,742 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:07,742 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. after waiting 0 ms 2023-02-28 04:57:07,742 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:07,742 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:07,742 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(1557): Region close journal for bd66d5b12692aefad5974a6eb15a4f47: 2023-02-28 04:57:07,742 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(7671): creating {ENCODED => 815cdd9dbbb8a2be3e8e0c8e96dd83ee, NAME => 'testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1677560227585', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:07,741 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:07,743 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(1603): Closing f60583096029c9557b712df1333409e7, disabling compactions & flushes 2023-02-28 04:57:07,743 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:07,743 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:07,744 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. after waiting 0 ms 2023-02-28 04:57:07,744 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:07,744 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:07,744 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(1557): Region close journal for f60583096029c9557b712df1333409e7: 2023-02-28 04:57:07,745 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(7671): creating {ENCODED => 17980904c451ad53a64d5259eea8aec2, NAME => 'testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1677560227585', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:07,745 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:07,746 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(1603): Closing 554a5845ea85f8afefc80a4e37e38fdc, disabling compactions & flushes 2023-02-28 04:57:07,746 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:07,746 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:07,746 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. after waiting 0 ms 2023-02-28 04:57:07,746 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:07,746 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:07,746 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(1557): Region close journal for 554a5845ea85f8afefc80a4e37e38fdc: 2023-02-28 04:57:07,746 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(7671): creating {ENCODED => e88d2578d5279f76f70975f8fd2b4155, NAME => 'testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1677560227585', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:07,771 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:07,772 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(1603): Closing 815cdd9dbbb8a2be3e8e0c8e96dd83ee, disabling compactions & flushes 2023-02-28 04:57:07,772 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:07,772 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:07,772 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. after waiting 0 ms 2023-02-28 04:57:07,774 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:07,774 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:07,774 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-0] regionserver.HRegion(1557): Region close journal for 815cdd9dbbb8a2be3e8e0c8e96dd83ee: 2023-02-28 04:57:07,776 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:07,776 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(1603): Closing 17980904c451ad53a64d5259eea8aec2, disabling compactions & flushes 2023-02-28 04:57:07,776 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:07,776 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:07,777 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. after waiting 0 ms 2023-02-28 04:57:07,777 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:07,777 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:07,777 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-1] regionserver.HRegion(1557): Region close journal for 17980904c451ad53a64d5259eea8aec2: 2023-02-28 04:57:07,777 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:07,777 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(1603): Closing e88d2578d5279f76f70975f8fd2b4155, disabling compactions & flushes 2023-02-28 04:57:07,777 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:07,777 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:07,777 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. after waiting 0 ms 2023-02-28 04:57:07,778 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:07,778 INFO [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:07,778 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1677560227585-pool-2] regionserver.HRegion(1557): Region close journal for e88d2578d5279f76f70975f8fd2b4155: 2023-02-28 04:57:07,782 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot-1677560227585 execute state=CREATE_TABLE_ADD_TO_META 2023-02-28 04:57:07,784 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560227784"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560227784"}]},"ts":"1677560227784"} 2023-02-28 04:57:07,784 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560227784"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560227784"}]},"ts":"1677560227784"} 2023-02-28 04:57:07,784 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560227784"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560227784"}]},"ts":"1677560227784"} 2023-02-28 04:57:07,784 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560227784"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560227784"}]},"ts":"1677560227784"} 2023-02-28 04:57:07,784 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560227784"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560227784"}]},"ts":"1677560227784"} 2023-02-28 04:57:07,785 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560227784"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560227784"}]},"ts":"1677560227784"} 2023-02-28 04:57:07,830 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-28 04:57:07,832 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot-1677560227585 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-28 04:57:07,832 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560227832"}]},"ts":"1677560227832"} 2023-02-28 04:57:07,834 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=ENABLING in hbase:meta 2023-02-28 04:57:07,840 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-28 04:57:07,845 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-28 04:57:07,845 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-28 04:57:07,845 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-28 04:57:07,845 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-28 04:57:07,846 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN}] 2023-02-28 04:57:07,850 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN 2023-02-28 04:57:07,850 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN 2023-02-28 04:57:07,852 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN 2023-02-28 04:57:07,852 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN 2023-02-28 04:57:07,854 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46629,1677560224813; forceNewPlan=false, retain=false 2023-02-28 04:57:07,854 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36597,1677560224857; forceNewPlan=false, retain=false 2023-02-28 04:57:07,854 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38247,1677560224887; forceNewPlan=false, retain=false 2023-02-28 04:57:07,855 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38247,1677560224887; forceNewPlan=false, retain=false 2023-02-28 04:57:07,855 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN 2023-02-28 04:57:07,856 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36597,1677560224857; forceNewPlan=false, retain=false 2023-02-28 04:57:07,856 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN 2023-02-28 04:57:07,858 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46629,1677560224813; forceNewPlan=false, retain=false 2023-02-28 04:57:07,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-28 04:57:08,004 INFO [jenkins-hbase4:42505] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-28 04:57:08,007 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:08,007 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:08,007 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:08,007 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560228007"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560228007"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560228007"}]},"ts":"1677560228007"} 2023-02-28 04:57:08,007 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:08,008 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560228007"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560228007"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560228007"}]},"ts":"1677560228007"} 2023-02-28 04:57:08,007 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:08,008 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560228007"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560228007"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560228007"}]},"ts":"1677560228007"} 2023-02-28 04:57:08,008 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560228007"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560228007"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560228007"}]},"ts":"1677560228007"} 2023-02-28 04:57:08,008 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560228007"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560228007"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560228007"}]},"ts":"1677560228007"} 2023-02-28 04:57:08,010 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=13, state=RUNNABLE; OpenRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:08,012 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=14, state=RUNNABLE; OpenRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:08,013 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=10, state=RUNNABLE; OpenRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:08,017 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=12, state=RUNNABLE; OpenRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:08,018 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:08,019 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560228018"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560228018"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560228018"}]},"ts":"1677560228018"} 2023-02-28 04:57:08,020 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=11, state=RUNNABLE; OpenRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:08,022 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=15, state=RUNNABLE; OpenRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:08,166 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:08,166 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-28 04:57:08,170 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55258, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-28 04:57:08,171 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:08,171 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-28 04:57:08,172 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:08,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 815cdd9dbbb8a2be3e8e0c8e96dd83ee, NAME => 'testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.', STARTKEY => '3', ENDKEY => '4'} 2023-02-28 04:57:08,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:08,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:08,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:08,173 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:08,175 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53346, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-28 04:57:08,177 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:08,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 17980904c451ad53a64d5259eea8aec2, NAME => 'testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.', STARTKEY => '4', ENDKEY => '5'} 2023-02-28 04:57:08,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:08,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:08,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:08,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:08,178 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:08,180 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:08,180 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:08,180 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf 2023-02-28 04:57:08,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e88d2578d5279f76f70975f8fd2b4155, NAME => 'testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.', STARTKEY => '5', ENDKEY => ''} 2023-02-28 04:57:08,181 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf 2023-02-28 04:57:08,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:08,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:08,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:08,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:08,182 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf 2023-02-28 04:57:08,182 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf 2023-02-28 04:57:08,183 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:08,185 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 815cdd9dbbb8a2be3e8e0c8e96dd83ee columnFamilyName cf 2023-02-28 04:57:08,185 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 17980904c451ad53a64d5259eea8aec2 columnFamilyName cf 2023-02-28 04:57:08,185 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf 2023-02-28 04:57:08,186 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf 2023-02-28 04:57:08,186 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e88d2578d5279f76f70975f8fd2b4155 columnFamilyName cf 2023-02-28 04:57:08,188 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(310): Store=815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:08,189 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(310): Store=17980904c451ad53a64d5259eea8aec2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:08,189 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(310): Store=e88d2578d5279f76f70975f8fd2b4155/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:08,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:08,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:08,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:08,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:08,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:08,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:08,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:08,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:08,198 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:08,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-28 04:57:08,204 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-28 04:57:08,204 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-28 04:57:08,204 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e88d2578d5279f76f70975f8fd2b4155; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74789537, jitterRate=0.11445094645023346}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:08,205 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 815cdd9dbbb8a2be3e8e0c8e96dd83ee; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64764849, jitterRate=-0.034928545355796814}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:08,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e88d2578d5279f76f70975f8fd2b4155: 2023-02-28 04:57:08,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 815cdd9dbbb8a2be3e8e0c8e96dd83ee: 2023-02-28 04:57:08,205 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 17980904c451ad53a64d5259eea8aec2; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69933533, jitterRate=0.042090848088264465}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:08,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 17980904c451ad53a64d5259eea8aec2: 2023-02-28 04:57:08,206 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee., pid=16, masterSystemTime=1677560228164 2023-02-28 04:57:08,206 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155., pid=21, masterSystemTime=1677560228171 2023-02-28 04:57:08,208 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2., pid=17, masterSystemTime=1677560228166 2023-02-28 04:57:08,211 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:08,211 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:08,211 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:08,211 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => bd66d5b12692aefad5974a6eb15a4f47, NAME => 'testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.', STARTKEY => '', ENDKEY => '1'} 2023-02-28 04:57:08,212 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:08,212 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:08,212 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:08,212 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:08,214 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:08,214 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560228213"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560228213"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560228213"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560228213"}]},"ts":"1677560228213"} 2023-02-28 04:57:08,214 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:08,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:08,216 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:08,216 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:08,216 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 554a5845ea85f8afefc80a4e37e38fdc, NAME => 'testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.', STARTKEY => '2', ENDKEY => '3'} 2023-02-28 04:57:08,216 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:08,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:08,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:08,217 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560228216"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560228216"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560228216"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560228216"}]},"ts":"1677560228216"} 2023-02-28 04:57:08,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:08,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:08,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:08,219 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:08,219 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:08,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f60583096029c9557b712df1333409e7, NAME => 'testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.', STARTKEY => '1', ENDKEY => '2'} 2023-02-28 04:57:08,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 f60583096029c9557b712df1333409e7 2023-02-28 04:57:08,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:08,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f60583096029c9557b712df1333409e7 2023-02-28 04:57:08,219 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f60583096029c9557b712df1333409e7 2023-02-28 04:57:08,220 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:08,220 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560228220"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560228220"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560228220"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560228220"}]},"ts":"1677560228220"} 2023-02-28 04:57:08,221 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf 2023-02-28 04:57:08,221 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf 2023-02-28 04:57:08,222 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bd66d5b12692aefad5974a6eb15a4f47 columnFamilyName cf 2023-02-28 04:57:08,224 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(310): Store=bd66d5b12692aefad5974a6eb15a4f47/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:08,226 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f60583096029c9557b712df1333409e7 2023-02-28 04:57:08,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:08,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:08,229 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:08,229 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf 2023-02-28 04:57:08,229 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf 2023-02-28 04:57:08,230 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f60583096029c9557b712df1333409e7 columnFamilyName cf 2023-02-28 04:57:08,230 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=13 2023-02-28 04:57:08,230 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=13, state=SUCCESS; OpenRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,38247,1677560224887 in 212 msec 2023-02-28 04:57:08,233 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf 2023-02-28 04:57:08,234 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(310): Store=f60583096029c9557b712df1333409e7/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:08,234 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf 2023-02-28 04:57:08,234 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:08,234 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=15 2023-02-28 04:57:08,234 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=15, state=SUCCESS; OpenRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,36597,1677560224857 in 201 msec 2023-02-28 04:57:08,234 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 554a5845ea85f8afefc80a4e37e38fdc columnFamilyName cf 2023-02-28 04:57:08,237 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(310): Store=554a5845ea85f8afefc80a4e37e38fdc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:08,237 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN in 385 msec 2023-02-28 04:57:08,237 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=14 2023-02-28 04:57:08,239 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=14, state=SUCCESS; OpenRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,46629,1677560224813 in 217 msec 2023-02-28 04:57:08,239 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN in 388 msec 2023-02-28 04:57:08,239 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:08,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-28 04:57:08,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:08,240 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:08,241 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened bd66d5b12692aefad5974a6eb15a4f47; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68319837, jitterRate=0.018044903874397278}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:08,241 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for bd66d5b12692aefad5974a6eb15a4f47: 2023-02-28 04:57:08,241 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:08,245 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN in 391 msec 2023-02-28 04:57:08,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47., pid=18, masterSystemTime=1677560228164 2023-02-28 04:57:08,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f60583096029c9557b712df1333409e7 2023-02-28 04:57:08,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:08,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:08,248 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:08,249 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:08,249 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560228248"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560228248"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560228248"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560228248"}]},"ts":"1677560228248"} 2023-02-28 04:57:08,253 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-28 04:57:08,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-28 04:57:08,255 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f60583096029c9557b712df1333409e7; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63735221, jitterRate=-0.05027119815349579}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:08,255 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f60583096029c9557b712df1333409e7: 2023-02-28 04:57:08,255 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 554a5845ea85f8afefc80a4e37e38fdc; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59441653, jitterRate=-0.11425034701824188}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:08,255 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 554a5845ea85f8afefc80a4e37e38fdc: 2023-02-28 04:57:08,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7., pid=20, masterSystemTime=1677560228166 2023-02-28 04:57:08,256 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=10 2023-02-28 04:57:08,256 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=10, state=SUCCESS; OpenRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,38247,1677560224887 in 238 msec 2023-02-28 04:57:08,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc., pid=19, masterSystemTime=1677560228171 2023-02-28 04:57:08,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:08,258 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:08,259 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN in 410 msec 2023-02-28 04:57:08,259 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:08,259 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560228259"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560228259"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560228259"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560228259"}]},"ts":"1677560228259"} 2023-02-28 04:57:08,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:08,261 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:08,261 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:08,262 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560228261"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560228261"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560228261"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560228261"}]},"ts":"1677560228261"} 2023-02-28 04:57:08,266 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=11 2023-02-28 04:57:08,267 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=11, state=SUCCESS; OpenRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,46629,1677560224813 in 243 msec 2023-02-28 04:57:08,268 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=12 2023-02-28 04:57:08,268 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=12, state=SUCCESS; OpenRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,36597,1677560224857 in 249 msec 2023-02-28 04:57:08,269 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN in 421 msec 2023-02-28 04:57:08,271 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=12, resume processing ppid=9 2023-02-28 04:57:08,271 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN in 422 msec 2023-02-28 04:57:08,272 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot-1677560227585 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-28 04:57:08,273 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560228272"}]},"ts":"1677560228272"} 2023-02-28 04:57:08,275 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=ENABLED in hbase:meta 2023-02-28 04:57:08,279 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1677560227585 execute state=CREATE_TABLE_POST_OPERATION 2023-02-28 04:57:08,282 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot-1677560227585 in 668 msec 2023-02-28 04:57:08,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-28 04:57:08,387 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot-1677560227585, procId: 9 completed 2023-02-28 04:57:08,393 DEBUG [Listener at localhost/36897] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1677560227585 2023-02-28 04:57:08,394 DEBUG [Listener at localhost/36897] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:08,397 INFO [Listener at localhost/36897] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-28 04:57:08,419 DEBUG [hconnection-0x56d8d559-shared-pool-5] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:08,423 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55264, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:08,425 DEBUG [hconnection-0x56d8d559-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:08,429 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53356, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:08,439 INFO [Listener at localhost/36897] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1677560227585 2023-02-28 04:57:08,443 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1677560227585 2023-02-28 04:57:08,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:08,454 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560228454"}]},"ts":"1677560228454"} 2023-02-28 04:57:08,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-28 04:57:08,456 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=DISABLING in hbase:meta 2023-02-28 04:57:08,458 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1677560227585 to state=DISABLING 2023-02-28 04:57:08,460 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN}, {pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN}, {pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN}, {pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN}, {pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN}, {pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN}] 2023-02-28 04:57:08,462 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN 2023-02-28 04:57:08,462 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN 2023-02-28 04:57:08,463 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN 2023-02-28 04:57:08,463 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN 2023-02-28 04:57:08,463 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN 2023-02-28 04:57:08,465 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:08,465 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:08,465 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:08,465 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560228465"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560228465"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560228465"}]},"ts":"1677560228465"} 2023-02-28 04:57:08,465 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560228465"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560228465"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560228465"}]},"ts":"1677560228465"} 2023-02-28 04:57:08,465 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560228465"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560228465"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560228465"}]},"ts":"1677560228465"} 2023-02-28 04:57:08,466 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:08,466 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560228466"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560228466"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560228466"}]},"ts":"1677560228466"} 2023-02-28 04:57:08,467 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:08,467 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560228466"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560228466"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560228466"}]},"ts":"1677560228466"} 2023-02-28 04:57:08,468 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=24, state=RUNNABLE; CloseRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:08,471 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=23, state=RUNNABLE; CloseRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:08,473 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=25, state=RUNNABLE; CloseRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:08,475 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=26, state=RUNNABLE; CloseRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:08,477 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=28, state=RUNNABLE; CloseRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:08,481 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN 2023-02-28 04:57:08,482 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:08,483 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560228482"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560228482"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560228482"}]},"ts":"1677560228482"} 2023-02-28 04:57:08,486 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=27, state=RUNNABLE; CloseRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:08,633 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:08,633 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f60583096029c9557b712df1333409e7 2023-02-28 04:57:08,633 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:08,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f60583096029c9557b712df1333409e7, disabling compactions & flushes 2023-02-28 04:57:08,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 815cdd9dbbb8a2be3e8e0c8e96dd83ee, disabling compactions & flushes 2023-02-28 04:57:08,634 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:08,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 554a5845ea85f8afefc80a4e37e38fdc, disabling compactions & flushes 2023-02-28 04:57:08,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:08,634 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:08,634 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:08,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:08,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. after waiting 0 ms 2023-02-28 04:57:08,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. after waiting 0 ms 2023-02-28 04:57:08,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:08,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:08,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:08,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. after waiting 0 ms 2023-02-28 04:57:08,634 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:08,641 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-28 04:57:08,641 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-28 04:57:08,644 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-28 04:57:08,644 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:08,644 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:08,644 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 554a5845ea85f8afefc80a4e37e38fdc: 2023-02-28 04:57:08,644 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f60583096029c9557b712df1333409e7: 2023-02-28 04:57:08,645 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:08,645 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 815cdd9dbbb8a2be3e8e0c8e96dd83ee: 2023-02-28 04:57:08,647 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f60583096029c9557b712df1333409e7 2023-02-28 04:57:08,647 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:08,647 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 17980904c451ad53a64d5259eea8aec2, disabling compactions & flushes 2023-02-28 04:57:08,647 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:08,647 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:08,647 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. after waiting 0 ms 2023-02-28 04:57:08,648 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:08,649 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=CLOSED 2023-02-28 04:57:08,649 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560228649"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560228649"}]},"ts":"1677560228649"} 2023-02-28 04:57:08,649 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:08,649 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:08,650 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e88d2578d5279f76f70975f8fd2b4155, disabling compactions & flushes 2023-02-28 04:57:08,650 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:08,650 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:08,650 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. after waiting 0 ms 2023-02-28 04:57:08,650 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:08,650 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=CLOSED 2023-02-28 04:57:08,651 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560228650"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560228650"}]},"ts":"1677560228650"} 2023-02-28 04:57:08,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:08,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:08,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing bd66d5b12692aefad5974a6eb15a4f47, disabling compactions & flushes 2023-02-28 04:57:08,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:08,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:08,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. after waiting 0 ms 2023-02-28 04:57:08,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:08,654 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=CLOSED 2023-02-28 04:57:08,654 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560228654"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560228654"}]},"ts":"1677560228654"} 2023-02-28 04:57:08,658 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=24 2023-02-28 04:57:08,659 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=24, state=SUCCESS; CloseRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,46629,1677560224813 in 187 msec 2023-02-28 04:57:08,660 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=25 2023-02-28 04:57:08,660 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=25, state=SUCCESS; CloseRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,36597,1677560224857 in 183 msec 2023-02-28 04:57:08,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-28 04:57:08,662 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=26 2023-02-28 04:57:08,662 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN in 198 msec 2023-02-28 04:57:08,662 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:08,662 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=26, state=SUCCESS; CloseRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,38247,1677560224887 in 182 msec 2023-02-28 04:57:08,663 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-28 04:57:08,663 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 17980904c451ad53a64d5259eea8aec2: 2023-02-28 04:57:08,663 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-28 04:57:08,664 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:08,664 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN in 200 msec 2023-02-28 04:57:08,664 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for bd66d5b12692aefad5974a6eb15a4f47: 2023-02-28 04:57:08,664 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:08,664 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e88d2578d5279f76f70975f8fd2b4155: 2023-02-28 04:57:08,665 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN in 202 msec 2023-02-28 04:57:08,665 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:08,666 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=CLOSED 2023-02-28 04:57:08,666 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560228666"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560228666"}]},"ts":"1677560228666"} 2023-02-28 04:57:08,666 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:08,667 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=CLOSED 2023-02-28 04:57:08,668 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560228667"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560228667"}]},"ts":"1677560228667"} 2023-02-28 04:57:08,670 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:08,671 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=CLOSED 2023-02-28 04:57:08,671 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560228670"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560228670"}]},"ts":"1677560228670"} 2023-02-28 04:57:08,671 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=27 2023-02-28 04:57:08,671 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=27, state=SUCCESS; CloseRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,46629,1677560224813 in 182 msec 2023-02-28 04:57:08,673 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=28 2023-02-28 04:57:08,673 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=28, state=SUCCESS; CloseRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,36597,1677560224857 in 193 msec 2023-02-28 04:57:08,674 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN in 211 msec 2023-02-28 04:57:08,675 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN in 213 msec 2023-02-28 04:57:08,676 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=23 2023-02-28 04:57:08,676 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=23, state=SUCCESS; CloseRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,38247,1677560224887 in 202 msec 2023-02-28 04:57:08,679 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=22 2023-02-28 04:57:08,679 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN in 216 msec 2023-02-28 04:57:08,680 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560228680"}]},"ts":"1677560228680"} 2023-02-28 04:57:08,682 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=DISABLED in hbase:meta 2023-02-28 04:57:08,685 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1677560227585 to state=DISABLED 2023-02-28 04:57:08,688 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1677560227585 in 242 msec 2023-02-28 04:57:08,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-28 04:57:08,707 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1677560227585, procId: 22 completed 2023-02-28 04:57:08,718 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:08,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1677560228718 (current time:1677560228718). 2023-02-28 04:57:08,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-28 04:57:08,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] zookeeper.ReadOnlyZKClient(139): Connect 0x02a40b10 to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:08,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4967516, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:08,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:08,734 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49978, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:08,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02a40b10 to 127.0.0.1:54355 2023-02-28 04:57:08,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:08,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-28 04:57:08,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:08,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:08,765 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:08,766 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=35, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE; timeout=600000, timestamp=1677560828766 2023-02-28 04:57:08,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:08,766 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1677560227585 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1677560227585 2023-02-28 04:57:08,772 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-28 04:57:08,772 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-28 04:57:08,792 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1677560227585 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } 2023-02-28 04:57:08,797 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:08,797 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:08,798 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:08,800 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:08,800 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:08,801 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:08,801 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:08,804 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:08,804 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:08,804 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:08,804 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:08,805 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:08,805 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:08,805 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:08,814 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:08,814 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:08,814 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:08,815 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:08,815 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:08,815 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:08,818 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:08,920 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1677560227585 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } as finished. 2023-02-28 04:57:08,921 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1677560227585 2023-02-28 04:57:08,923 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/emptySnaptb-1677560227585 2023-02-28 04:57:08,969 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/emptySnaptb-1677560227585 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/emptySnaptb-1677560227585 2023-02-28 04:57:09,004 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1677560227585 of table testRestoreSnapshot-1677560227585 completed 2023-02-28 04:57:09,005 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/emptySnaptb-1677560227585 2023-02-28 04:57:09,005 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/emptySnaptb-1677560227585 2023-02-28 04:57:09,005 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1677560227585 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1677560227585 at 1677560228767Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1677560227585 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } at 1677560228792 (+25 ms)Consolidate snapshot: emptySnaptb-1677560227585 at 1677560228921 (+129 ms)Loading Region manifests for emptySnaptb-1677560227585 at 1677560228921Writing data manifest for emptySnaptb-1677560227585 at 1677560228940 (+19 ms)Verifying snapshot: emptySnaptb-1677560227585 at 1677560228955 (+15 ms)Snapshot emptySnaptb-1677560227585 of table testRestoreSnapshot-1677560227585 completed at 1677560229004 (+49 ms) 2023-02-28 04:57:09,007 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=35, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:09,009 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE in 253 msec 2023-02-28 04:57:09,022 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-28 04:57:09,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } is done 2023-02-28 04:57:09,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-28 04:57:09,029 INFO [Listener at localhost/36897] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1677560227585 2023-02-28 04:57:09,034 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1677560227585 2023-02-28 04:57:09,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=36, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:09,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-28 04:57:09,045 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1677560227585 2023-02-28 04:57:09,045 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560229045"}]},"ts":"1677560229045"} 2023-02-28 04:57:09,047 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=ENABLING in hbase:meta 2023-02-28 04:57:09,051 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-28 04:57:09,053 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN}, {pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN}, {pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN}, {pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN}, {pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN}, {pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN}] 2023-02-28 04:57:09,056 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN 2023-02-28 04:57:09,056 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN 2023-02-28 04:57:09,057 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN 2023-02-28 04:57:09,057 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN 2023-02-28 04:57:09,058 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:09,058 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:09,058 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:09,058 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN 2023-02-28 04:57:09,059 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,36597,1677560224857; forceNewPlan=false, retain=true 2023-02-28 04:57:09,058 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:09,058 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,38247,1677560224887; forceNewPlan=false, retain=true 2023-02-28 04:57:09,058 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46629,1677560224813; forceNewPlan=false, retain=true 2023-02-28 04:57:09,059 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,36597,1677560224857; forceNewPlan=false, retain=true 2023-02-28 04:57:09,060 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:09,060 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN 2023-02-28 04:57:09,060 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46629,1677560224813; forceNewPlan=false, retain=true 2023-02-28 04:57:09,062 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:09,062 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,38247,1677560224887; forceNewPlan=false, retain=true 2023-02-28 04:57:09,209 INFO [jenkins-hbase4:42505] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-28 04:57:09,209 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:09,209 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:09,209 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:09,209 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560229209"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560229209"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560229209"}]},"ts":"1677560229209"} 2023-02-28 04:57:09,209 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:09,209 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:09,210 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560229209"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560229209"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560229209"}]},"ts":"1677560229209"} 2023-02-28 04:57:09,210 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560229209"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560229209"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560229209"}]},"ts":"1677560229209"} 2023-02-28 04:57:09,210 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560229209"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560229209"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560229209"}]},"ts":"1677560229209"} 2023-02-28 04:57:09,210 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560229209"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560229209"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560229209"}]},"ts":"1677560229209"} 2023-02-28 04:57:09,212 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=41, state=RUNNABLE; OpenRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:09,213 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=39, state=RUNNABLE; OpenRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:09,214 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=40, state=RUNNABLE; OpenRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:09,216 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=38, state=RUNNABLE; OpenRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:09,217 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:09,217 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=37, state=RUNNABLE; OpenRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:09,217 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560229217"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560229217"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560229217"}]},"ts":"1677560229217"} 2023-02-28 04:57:09,219 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=42, state=RUNNABLE; OpenRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:09,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-28 04:57:09,369 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:09,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f60583096029c9557b712df1333409e7, NAME => 'testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.', STARTKEY => '1', ENDKEY => '2'} 2023-02-28 04:57:09,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:09,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e88d2578d5279f76f70975f8fd2b4155, NAME => 'testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.', STARTKEY => '5', ENDKEY => ''} 2023-02-28 04:57:09,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 f60583096029c9557b712df1333409e7 2023-02-28 04:57:09,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:09,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f60583096029c9557b712df1333409e7 2023-02-28 04:57:09,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f60583096029c9557b712df1333409e7 2023-02-28 04:57:09,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:09,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:09,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:09,370 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:09,371 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:09,371 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => bd66d5b12692aefad5974a6eb15a4f47, NAME => 'testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.', STARTKEY => '', ENDKEY => '1'} 2023-02-28 04:57:09,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:09,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:09,372 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f60583096029c9557b712df1333409e7 2023-02-28 04:57:09,372 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:09,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:09,372 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:09,373 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf 2023-02-28 04:57:09,373 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf 2023-02-28 04:57:09,374 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf 2023-02-28 04:57:09,374 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf 2023-02-28 04:57:09,374 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:09,374 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f60583096029c9557b712df1333409e7 columnFamilyName cf 2023-02-28 04:57:09,375 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf 2023-02-28 04:57:09,375 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e88d2578d5279f76f70975f8fd2b4155 columnFamilyName cf 2023-02-28 04:57:09,376 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(310): Store=f60583096029c9557b712df1333409e7/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:09,375 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf 2023-02-28 04:57:09,377 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bd66d5b12692aefad5974a6eb15a4f47 columnFamilyName cf 2023-02-28 04:57:09,378 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(310): Store=e88d2578d5279f76f70975f8fd2b4155/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:09,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:09,379 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(310): Store=bd66d5b12692aefad5974a6eb15a4f47/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:09,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:09,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:09,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:09,382 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:09,383 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:09,385 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f60583096029c9557b712df1333409e7 2023-02-28 04:57:09,386 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:09,387 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:09,387 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f60583096029c9557b712df1333409e7; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71308110, jitterRate=0.06257364153862}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:09,387 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f60583096029c9557b712df1333409e7: 2023-02-28 04:57:09,387 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e88d2578d5279f76f70975f8fd2b4155; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60007939, jitterRate=-0.10581202805042267}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:09,387 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e88d2578d5279f76f70975f8fd2b4155: 2023-02-28 04:57:09,388 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155., pid=48, masterSystemTime=1677560229365 2023-02-28 04:57:09,388 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7., pid=46, masterSystemTime=1677560229364 2023-02-28 04:57:09,389 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened bd66d5b12692aefad5974a6eb15a4f47; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68016204, jitterRate=0.013520419597625732}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:09,389 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for bd66d5b12692aefad5974a6eb15a4f47: 2023-02-28 04:57:09,390 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47., pid=47, masterSystemTime=1677560229368 2023-02-28 04:57:09,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:09,391 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:09,391 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:09,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 17980904c451ad53a64d5259eea8aec2, NAME => 'testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.', STARTKEY => '4', ENDKEY => '5'} 2023-02-28 04:57:09,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:09,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:09,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:09,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:09,392 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:09,392 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560229392"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560229392"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560229392"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560229392"}]},"ts":"1677560229392"} 2023-02-28 04:57:09,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:09,393 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:09,394 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:09,394 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 554a5845ea85f8afefc80a4e37e38fdc, NAME => 'testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.', STARTKEY => '2', ENDKEY => '3'} 2023-02-28 04:57:09,394 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:09,394 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:09,394 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:09,394 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:09,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:09,395 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:09,396 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560229395"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560229395"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560229395"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560229395"}]},"ts":"1677560229395"} 2023-02-28 04:57:09,396 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:09,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:09,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:09,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 815cdd9dbbb8a2be3e8e0c8e96dd83ee, NAME => 'testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.', STARTKEY => '3', ENDKEY => '4'} 2023-02-28 04:57:09,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:09,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:09,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:09,397 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:09,397 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:09,400 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560229397"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560229397"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560229397"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560229397"}]},"ts":"1677560229397"} 2023-02-28 04:57:09,400 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf 2023-02-28 04:57:09,400 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf 2023-02-28 04:57:09,401 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:09,401 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 17980904c451ad53a64d5259eea8aec2 columnFamilyName cf 2023-02-28 04:57:09,402 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:09,402 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf 2023-02-28 04:57:09,402 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf 2023-02-28 04:57:09,403 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(310): Store=17980904c451ad53a64d5259eea8aec2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:09,403 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 554a5845ea85f8afefc80a4e37e38fdc columnFamilyName cf 2023-02-28 04:57:09,403 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=38 2023-02-28 04:57:09,403 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=38, state=SUCCESS; OpenRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,46629,1677560224813 in 180 msec 2023-02-28 04:57:09,404 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf 2023-02-28 04:57:09,404 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf 2023-02-28 04:57:09,404 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(310): Store=554a5845ea85f8afefc80a4e37e38fdc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:09,405 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 815cdd9dbbb8a2be3e8e0c8e96dd83ee columnFamilyName cf 2023-02-28 04:57:09,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:09,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:09,406 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=42 2023-02-28 04:57:09,406 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=42, state=SUCCESS; OpenRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,36597,1677560224857 in 183 msec 2023-02-28 04:57:09,407 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:09,407 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(310): Store=815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:09,407 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN in 352 msec 2023-02-28 04:57:09,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:09,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:09,410 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=37 2023-02-28 04:57:09,410 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=37, state=SUCCESS; OpenRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,38247,1677560224887 in 188 msec 2023-02-28 04:57:09,410 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN in 355 msec 2023-02-28 04:57:09,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:09,413 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:09,413 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:09,414 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 17980904c451ad53a64d5259eea8aec2; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63846809, jitterRate=-0.048608407378196716}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:09,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 17980904c451ad53a64d5259eea8aec2: 2023-02-28 04:57:09,415 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 554a5845ea85f8afefc80a4e37e38fdc; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67319716, jitterRate=0.003141939640045166}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:09,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 554a5845ea85f8afefc80a4e37e38fdc: 2023-02-28 04:57:09,416 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2., pid=43, masterSystemTime=1677560229364 2023-02-28 04:57:09,416 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc., pid=44, masterSystemTime=1677560229365 2023-02-28 04:57:09,416 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN in 359 msec 2023-02-28 04:57:09,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:09,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:09,418 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:09,418 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 815cdd9dbbb8a2be3e8e0c8e96dd83ee; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70500084, jitterRate=0.05053311586380005}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:09,419 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 815cdd9dbbb8a2be3e8e0c8e96dd83ee: 2023-02-28 04:57:09,419 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:09,419 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560229419"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560229419"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560229419"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560229419"}]},"ts":"1677560229419"} 2023-02-28 04:57:09,419 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee., pid=45, masterSystemTime=1677560229368 2023-02-28 04:57:09,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:09,420 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:09,421 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:09,421 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560229421"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560229421"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560229421"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560229421"}]},"ts":"1677560229421"} 2023-02-28 04:57:09,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:09,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:09,422 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:09,422 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560229422"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560229422"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560229422"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560229422"}]},"ts":"1677560229422"} 2023-02-28 04:57:09,426 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=39 2023-02-28 04:57:09,426 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=39, state=SUCCESS; OpenRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,36597,1677560224857 in 209 msec 2023-02-28 04:57:09,427 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=41 2023-02-28 04:57:09,427 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=41, state=SUCCESS; OpenRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,46629,1677560224813 in 212 msec 2023-02-28 04:57:09,429 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN in 375 msec 2023-02-28 04:57:09,429 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=40 2023-02-28 04:57:09,429 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=40, state=SUCCESS; OpenRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,38247,1677560224887 in 211 msec 2023-02-28 04:57:09,430 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN in 376 msec 2023-02-28 04:57:09,432 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=36 2023-02-28 04:57:09,432 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN in 378 msec 2023-02-28 04:57:09,433 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560229433"}]},"ts":"1677560229433"} 2023-02-28 04:57:09,435 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=ENABLED in hbase:meta 2023-02-28 04:57:09,437 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1677560227585' was successfully enabled. 2023-02-28 04:57:09,440 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=36, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1677560227585 in 403 msec 2023-02-28 04:57:09,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-28 04:57:09,798 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1677560227585, procId: 36 completed 2023-02-28 04:57:09,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:09,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:09,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:09,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:09,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:09,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:09,857 DEBUG [Listener at localhost/36897] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1677560227585 2023-02-28 04:57:09,857 DEBUG [Listener at localhost/36897] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:09,858 INFO [Listener at localhost/36897] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-28 04:57:09,879 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-28 04:57:09,881 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-28 04:57:09,883 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-28 04:57:09,885 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-28 04:57:09,887 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-28 04:57:09,897 INFO [Listener at localhost/36897] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1677560227585 2023-02-28 04:57:09,897 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1677560227585 2023-02-28 04:57:09,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:09,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-28 04:57:09,902 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560229902"}]},"ts":"1677560229902"} 2023-02-28 04:57:09,904 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=DISABLING in hbase:meta 2023-02-28 04:57:09,906 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1677560227585 to state=DISABLING 2023-02-28 04:57:09,907 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN}, {pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN}, {pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN}, {pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN}, {pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN}, {pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN}] 2023-02-28 04:57:09,909 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN 2023-02-28 04:57:09,909 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN 2023-02-28 04:57:09,910 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN 2023-02-28 04:57:09,910 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN 2023-02-28 04:57:09,910 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN 2023-02-28 04:57:09,911 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:09,911 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560229911"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560229911"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560229911"}]},"ts":"1677560229911"} 2023-02-28 04:57:09,911 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:09,911 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:09,911 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:09,911 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560229911"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560229911"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560229911"}]},"ts":"1677560229911"} 2023-02-28 04:57:09,911 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560229911"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560229911"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560229911"}]},"ts":"1677560229911"} 2023-02-28 04:57:09,911 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560229911"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560229911"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560229911"}]},"ts":"1677560229911"} 2023-02-28 04:57:09,912 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:09,912 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560229911"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560229911"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560229911"}]},"ts":"1677560229911"} 2023-02-28 04:57:09,913 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=51, state=RUNNABLE; CloseRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:09,914 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=53, state=RUNNABLE; CloseRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:09,915 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=50, state=RUNNABLE; CloseRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:09,916 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=52, state=RUNNABLE; CloseRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:09,918 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=55, state=RUNNABLE; CloseRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:09,919 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN 2023-02-28 04:57:09,920 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:09,920 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560229920"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560229920"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560229920"}]},"ts":"1677560229920"} 2023-02-28 04:57:09,922 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=54, state=RUNNABLE; CloseRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:10,066 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f60583096029c9557b712df1333409e7 2023-02-28 04:57:10,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f60583096029c9557b712df1333409e7, disabling compactions & flushes 2023-02-28 04:57:10,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:10,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:10,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. after waiting 0 ms 2023-02-28 04:57:10,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:10,068 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:10,068 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing f60583096029c9557b712df1333409e7 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-28 04:57:10,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing bd66d5b12692aefad5974a6eb15a4f47, disabling compactions & flushes 2023-02-28 04:57:10,068 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:10,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:10,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. after waiting 0 ms 2023-02-28 04:57:10,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:10,068 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing bd66d5b12692aefad5974a6eb15a4f47 1/1 column families, dataSize=2.42 KB heapSize=5.45 KB 2023-02-28 04:57:10,071 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:10,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 554a5845ea85f8afefc80a4e37e38fdc, disabling compactions & flushes 2023-02-28 04:57:10,072 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:10,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:10,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. after waiting 0 ms 2023-02-28 04:57:10,072 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:10,072 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 554a5845ea85f8afefc80a4e37e38fdc 1/1 column families, dataSize=2.16 KB heapSize=4.89 KB 2023-02-28 04:57:10,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-28 04:57:10,189 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 2023-02-28 04:57:10,189 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 2023-02-28 04:57:10,189 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b 2023-02-28 04:57:10,223 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/.tmp/cf/7c2abaa2c369456bb51f5b5e910c2efc 2023-02-28 04:57:10,223 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/.tmp/cf/ffded11160e44993a6ff131b4e9f7276 2023-02-28 04:57:10,231 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/.tmp/cf/15b46826798648acb28930b8ad9e7857 2023-02-28 04:57:10,233 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/.tmp/cf/ffded11160e44993a6ff131b4e9f7276 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/ffded11160e44993a6ff131b4e9f7276 2023-02-28 04:57:10,233 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/.tmp/cf/7c2abaa2c369456bb51f5b5e910c2efc as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/7c2abaa2c369456bb51f5b5e910c2efc 2023-02-28 04:57:10,241 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/.tmp/cf/15b46826798648acb28930b8ad9e7857 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/15b46826798648acb28930b8ad9e7857 2023-02-28 04:57:10,242 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/7c2abaa2c369456bb51f5b5e910c2efc, entries=37, sequenceid=8, filesize=11.2 K 2023-02-28 04:57:10,242 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/ffded11160e44993a6ff131b4e9f7276, entries=33, sequenceid=8, filesize=10.5 K 2023-02-28 04:57:10,246 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.42 KB/2477, heapSize ~5.44 KB/5568, currentSize=0 B/0 for bd66d5b12692aefad5974a6eb15a4f47 in 178ms, sequenceid=8, compaction requested=false 2023-02-28 04:57:10,248 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1677560227585' 2023-02-28 04:57:10,253 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/15b46826798648acb28930b8ad9e7857, entries=28, sequenceid=8, filesize=9.7 K 2023-02-28 04:57:10,253 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.16 KB/2209, heapSize ~4.88 KB/4992, currentSize=0 B/0 for 554a5845ea85f8afefc80a4e37e38fdc in 181ms, sequenceid=8, compaction requested=false 2023-02-28 04:57:10,256 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for f60583096029c9557b712df1333409e7 in 188ms, sequenceid=8, compaction requested=false 2023-02-28 04:57:10,261 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-28 04:57:10,261 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:10,262 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for bd66d5b12692aefad5974a6eb15a4f47: 2023-02-28 04:57:10,262 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-28 04:57:10,262 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-28 04:57:10,262 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:10,262 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 554a5845ea85f8afefc80a4e37e38fdc: 2023-02-28 04:57:10,263 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:10,263 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f60583096029c9557b712df1333409e7: 2023-02-28 04:57:10,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:10,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:10,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 815cdd9dbbb8a2be3e8e0c8e96dd83ee, disabling compactions & flushes 2023-02-28 04:57:10,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:10,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:10,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. after waiting 0 ms 2023-02-28 04:57:10,264 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:10,264 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 815cdd9dbbb8a2be3e8e0c8e96dd83ee 1/1 column families, dataSize=2.29 KB heapSize=5.17 KB 2023-02-28 04:57:10,264 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=CLOSED 2023-02-28 04:57:10,264 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560230264"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560230264"}]},"ts":"1677560230264"} 2023-02-28 04:57:10,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:10,266 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:10,266 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=CLOSED 2023-02-28 04:57:10,267 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560230266"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560230266"}]},"ts":"1677560230266"} 2023-02-28 04:57:10,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e88d2578d5279f76f70975f8fd2b4155, disabling compactions & flushes 2023-02-28 04:57:10,267 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f60583096029c9557b712df1333409e7 2023-02-28 04:57:10,267 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:10,267 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:10,268 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 17980904c451ad53a64d5259eea8aec2, disabling compactions & flushes 2023-02-28 04:57:10,268 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:10,268 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:10,268 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=CLOSED 2023-02-28 04:57:10,269 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560230268"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560230268"}]},"ts":"1677560230268"} 2023-02-28 04:57:10,268 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. after waiting 0 ms 2023-02-28 04:57:10,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:10,268 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:10,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing e88d2578d5279f76f70975f8fd2b4155 1/1 column families, dataSize=22.31 KB heapSize=48.20 KB 2023-02-28 04:57:10,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. after waiting 0 ms 2023-02-28 04:57:10,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:10,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 17980904c451ad53a64d5259eea8aec2 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-28 04:57:10,281 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=50 2023-02-28 04:57:10,281 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=50, state=SUCCESS; CloseRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,38247,1677560224887 in 354 msec 2023-02-28 04:57:10,282 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=52 2023-02-28 04:57:10,282 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=52, state=SUCCESS; CloseRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,36597,1677560224857 in 355 msec 2023-02-28 04:57:10,283 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=51 2023-02-28 04:57:10,283 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=51, state=SUCCESS; CloseRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,46629,1677560224813 in 361 msec 2023-02-28 04:57:10,284 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN in 374 msec 2023-02-28 04:57:10,285 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN in 375 msec 2023-02-28 04:57:10,285 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN in 376 msec 2023-02-28 04:57:10,314 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 2023-02-28 04:57:10,330 WARN [DataStreamer for file /user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-02-28 04:57:10,331 WARN [DataStreamer for file /user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/.tmp/cf/b4ee3e4950594313bbb87aa56d665751] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-02-28 04:57:10,332 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 2023-02-28 04:57:10,335 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/.tmp/cf/b4ee3e4950594313bbb87aa56d665751 2023-02-28 04:57:10,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 2023-02-28 04:57:10,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/.tmp/cf/b4ee3e4950594313bbb87aa56d665751 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/b4ee3e4950594313bbb87aa56d665751 2023-02-28 04:57:10,350 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/.tmp/cf/f336080ab5da417c9ad6f84197eeb6c5 2023-02-28 04:57:10,354 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/b4ee3e4950594313bbb87aa56d665751, entries=35, sequenceid=8, filesize=10.9 K 2023-02-28 04:57:10,358 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.29 KB/2343, heapSize ~5.16 KB/5280, currentSize=0 B/0 for 815cdd9dbbb8a2be3e8e0c8e96dd83ee in 93ms, sequenceid=8, compaction requested=false 2023-02-28 04:57:10,361 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/.tmp/cf/7320fea4678244c4b9b33f44fe5a0756 2023-02-28 04:57:10,364 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/.tmp/cf/f336080ab5da417c9ad6f84197eeb6c5 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/f336080ab5da417c9ad6f84197eeb6c5 2023-02-28 04:57:10,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-28 04:57:10,366 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:10,366 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 815cdd9dbbb8a2be3e8e0c8e96dd83ee: 2023-02-28 04:57:10,369 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:10,369 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/.tmp/cf/7320fea4678244c4b9b33f44fe5a0756 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/7320fea4678244c4b9b33f44fe5a0756 2023-02-28 04:57:10,370 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=CLOSED 2023-02-28 04:57:10,370 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560230370"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560230370"}]},"ts":"1677560230370"} 2023-02-28 04:57:10,372 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/f336080ab5da417c9ad6f84197eeb6c5, entries=26, sequenceid=8, filesize=9.3 K 2023-02-28 04:57:10,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for 17980904c451ad53a64d5259eea8aec2 in 104ms, sequenceid=8, compaction requested=false 2023-02-28 04:57:10,377 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=53 2023-02-28 04:57:10,377 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=53, state=SUCCESS; CloseRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,38247,1677560224887 in 458 msec 2023-02-28 04:57:10,380 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN in 470 msec 2023-02-28 04:57:10,380 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/7320fea4678244c4b9b33f44fe5a0756, entries=341, sequenceid=8, filesize=63.0 K 2023-02-28 04:57:10,381 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.31 KB/22843, heapSize ~48.19 KB/49344, currentSize=0 B/0 for e88d2578d5279f76f70975f8fd2b4155 in 112ms, sequenceid=8, compaction requested=false 2023-02-28 04:57:10,385 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-28 04:57:10,386 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:10,386 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 17980904c451ad53a64d5259eea8aec2: 2023-02-28 04:57:10,388 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:10,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-28 04:57:10,389 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=CLOSED 2023-02-28 04:57:10,389 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560230389"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560230389"}]},"ts":"1677560230389"} 2023-02-28 04:57:10,389 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:10,389 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e88d2578d5279f76f70975f8fd2b4155: 2023-02-28 04:57:10,391 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:10,392 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=CLOSED 2023-02-28 04:57:10,392 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560230392"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560230392"}]},"ts":"1677560230392"} 2023-02-28 04:57:10,393 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=54 2023-02-28 04:57:10,393 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=54, state=SUCCESS; CloseRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,46629,1677560224813 in 469 msec 2023-02-28 04:57:10,395 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN in 486 msec 2023-02-28 04:57:10,396 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=55 2023-02-28 04:57:10,396 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=55, state=SUCCESS; CloseRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,36597,1677560224857 in 476 msec 2023-02-28 04:57:10,399 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=49 2023-02-28 04:57:10,399 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN in 489 msec 2023-02-28 04:57:10,400 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560230399"}]},"ts":"1677560230399"} 2023-02-28 04:57:10,401 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=DISABLED in hbase:meta 2023-02-28 04:57:10,403 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1677560227585 to state=DISABLED 2023-02-28 04:57:10,405 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1677560227585 in 506 msec 2023-02-28 04:57:10,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-28 04:57:10,655 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1677560227585, procId: 49 completed 2023-02-28 04:57:10,656 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:10,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1677560230656 (current time:1677560230656). 2023-02-28 04:57:10,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-28 04:57:10,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] zookeeper.ReadOnlyZKClient(139): Connect 0x2f213cb8 to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:10,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dbca83e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:10,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:10,665 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49992, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:10,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2f213cb8 to 127.0.0.1:54355 2023-02-28 04:57:10,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:10,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-28 04:57:10,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:10,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:10,670 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:10,671 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=62, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE; timeout=600000, timestamp=1677560830671 2023-02-28 04:57:10,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:10,671 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1677560227585 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1677560227585 2023-02-28 04:57:10,672 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-28 04:57:10,672 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-28 04:57:10,687 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1677560227585 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } 2023-02-28 04:57:10,689 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:10,689 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:10,689 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:10,689 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:10,689 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:10,689 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:10,690 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:10,690 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:10,690 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:10,690 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:10,690 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:10,690 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:10,690 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:10,690 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:10,693 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/b4ee3e4950594313bbb87aa56d665751] hfiles 2023-02-28 04:57:10,693 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/b4ee3e4950594313bbb87aa56d665751 2023-02-28 04:57:10,693 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/15b46826798648acb28930b8ad9e7857] hfiles 2023-02-28 04:57:10,693 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/15b46826798648acb28930b8ad9e7857 2023-02-28 04:57:10,695 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/7320fea4678244c4b9b33f44fe5a0756] hfiles 2023-02-28 04:57:10,695 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/7320fea4678244c4b9b33f44fe5a0756 2023-02-28 04:57:10,696 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/f336080ab5da417c9ad6f84197eeb6c5] hfiles 2023-02-28 04:57:10,697 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/ffded11160e44993a6ff131b4e9f7276] hfiles 2023-02-28 04:57:10,697 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/ffded11160e44993a6ff131b4e9f7276 2023-02-28 04:57:10,697 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/7c2abaa2c369456bb51f5b5e910c2efc] hfiles 2023-02-28 04:57:10,697 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/f336080ab5da417c9ad6f84197eeb6c5 2023-02-28 04:57:10,697 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/7c2abaa2c369456bb51f5b5e910c2efc 2023-02-28 04:57:10,698 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963] hfiles 2023-02-28 04:57:10,698 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 2023-02-28 04:57:10,698 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 2023-02-28 04:57:10,698 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 2023-02-28 04:57:10,698 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b 2023-02-28 04:57:10,699 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 2023-02-28 04:57:10,699 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 2023-02-28 04:57:10,746 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1677560227585 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } as finished. 2023-02-28 04:57:10,746 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1677560227585 2023-02-28 04:57:10,747 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/snaptb0-1677560227585 2023-02-28 04:57:10,796 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/snaptb0-1677560227585 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/snaptb0-1677560227585 2023-02-28 04:57:10,821 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1677560227585 of table testRestoreSnapshot-1677560227585 completed 2023-02-28 04:57:10,822 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/snaptb0-1677560227585 2023-02-28 04:57:10,822 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/snaptb0-1677560227585 2023-02-28 04:57:10,822 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1677560227585 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1677560227585 at 1677560230671Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1677560227585 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } at 1677560230687 (+16 ms)Consolidate snapshot: snaptb0-1677560227585 at 1677560230746 (+59 ms)Loading Region manifests for snaptb0-1677560227585 at 1677560230746Writing data manifest for snaptb0-1677560227585 at 1677560230764 (+18 ms)Verifying snapshot: snaptb0-1677560227585 at 1677560230778 (+14 ms)Snapshot snaptb0-1677560227585 of table testRestoreSnapshot-1677560227585 completed at 1677560230821 (+43 ms) 2023-02-28 04:57:10,823 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=62, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:10,825 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE in 154 msec 2023-02-28 04:57:10,922 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-28 04:57:10,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } is done 2023-02-28 04:57:10,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-28 04:57:10,924 INFO [Listener at localhost/36897] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1677560227585 2023-02-28 04:57:10,925 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1677560227585 2023-02-28 04:57:10,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:10,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-28 04:57:10,929 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1677560227585 2023-02-28 04:57:10,929 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560230929"}]},"ts":"1677560230929"} 2023-02-28 04:57:10,931 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=ENABLING in hbase:meta 2023-02-28 04:57:10,938 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-28 04:57:10,938 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN}, {pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN}, {pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN}, {pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN}, {pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN}] 2023-02-28 04:57:10,940 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN 2023-02-28 04:57:10,940 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN 2023-02-28 04:57:10,940 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN 2023-02-28 04:57:10,940 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN 2023-02-28 04:57:10,941 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN 2023-02-28 04:57:10,941 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:10,941 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:10,941 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46629,1677560224813; forceNewPlan=false, retain=true 2023-02-28 04:57:10,941 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:10,941 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:10,941 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,38247,1677560224887; forceNewPlan=false, retain=true 2023-02-28 04:57:10,941 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,36597,1677560224857; forceNewPlan=false, retain=true 2023-02-28 04:57:10,941 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,36597,1677560224857; forceNewPlan=false, retain=true 2023-02-28 04:57:10,942 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:10,943 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46629,1677560224813; forceNewPlan=false, retain=true 2023-02-28 04:57:10,943 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN 2023-02-28 04:57:10,943 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:10,944 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,38247,1677560224887; forceNewPlan=false, retain=true 2023-02-28 04:57:11,091 INFO [jenkins-hbase4:42505] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-28 04:57:11,092 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:11,092 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:11,092 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:11,092 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:11,092 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560231092"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560231092"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560231092"}]},"ts":"1677560231092"} 2023-02-28 04:57:11,092 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:11,092 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560231092"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560231092"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560231092"}]},"ts":"1677560231092"} 2023-02-28 04:57:11,092 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560231092"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560231092"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560231092"}]},"ts":"1677560231092"} 2023-02-28 04:57:11,092 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560231092"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560231092"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560231092"}]},"ts":"1677560231092"} 2023-02-28 04:57:11,092 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560231092"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560231092"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560231092"}]},"ts":"1677560231092"} 2023-02-28 04:57:11,094 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=67, state=RUNNABLE; OpenRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:11,095 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=64, state=RUNNABLE; OpenRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:11,096 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=65, state=RUNNABLE; OpenRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:11,097 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=66, state=RUNNABLE; OpenRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:11,098 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=68, state=RUNNABLE; OpenRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:11,099 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:11,099 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560231099"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560231099"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560231099"}]},"ts":"1677560231099"} 2023-02-28 04:57:11,107 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=69, state=RUNNABLE; OpenRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:11,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-28 04:57:11,250 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:11,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 815cdd9dbbb8a2be3e8e0c8e96dd83ee, NAME => 'testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.', STARTKEY => '3', ENDKEY => '4'} 2023-02-28 04:57:11,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:11,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:11,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:11,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:11,253 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:11,253 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:11,253 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f60583096029c9557b712df1333409e7, NAME => 'testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.', STARTKEY => '1', ENDKEY => '2'} 2023-02-28 04:57:11,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 f60583096029c9557b712df1333409e7 2023-02-28 04:57:11,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:11,254 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:11,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f60583096029c9557b712df1333409e7 2023-02-28 04:57:11,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f60583096029c9557b712df1333409e7 2023-02-28 04:57:11,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e88d2578d5279f76f70975f8fd2b4155, NAME => 'testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.', STARTKEY => '5', ENDKEY => ''} 2023-02-28 04:57:11,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:11,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:11,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:11,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:11,255 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf 2023-02-28 04:57:11,255 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf 2023-02-28 04:57:11,255 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 815cdd9dbbb8a2be3e8e0c8e96dd83ee columnFamilyName cf 2023-02-28 04:57:11,256 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:11,258 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf 2023-02-28 04:57:11,258 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf 2023-02-28 04:57:11,258 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e88d2578d5279f76f70975f8fd2b4155 columnFamilyName cf 2023-02-28 04:57:11,260 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f60583096029c9557b712df1333409e7 2023-02-28 04:57:11,261 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf 2023-02-28 04:57:11,261 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf 2023-02-28 04:57:11,261 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f60583096029c9557b712df1333409e7 columnFamilyName cf 2023-02-28 04:57:11,267 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/b4ee3e4950594313bbb87aa56d665751 2023-02-28 04:57:11,268 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(310): Store=815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:11,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:11,272 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/15b46826798648acb28930b8ad9e7857 2023-02-28 04:57:11,272 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(310): Store=f60583096029c9557b712df1333409e7/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:11,272 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/7320fea4678244c4b9b33f44fe5a0756 2023-02-28 04:57:11,273 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(310): Store=e88d2578d5279f76f70975f8fd2b4155/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:11,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:11,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:11,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:11,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:11,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:11,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:11,278 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 815cdd9dbbb8a2be3e8e0c8e96dd83ee; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74350477, jitterRate=0.10790844261646271}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:11,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 815cdd9dbbb8a2be3e8e0c8e96dd83ee: 2023-02-28 04:57:11,279 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f60583096029c9557b712df1333409e7 2023-02-28 04:57:11,280 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee., pid=70, masterSystemTime=1677560231246 2023-02-28 04:57:11,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f60583096029c9557b712df1333409e7; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72417524, jitterRate=0.0791051983833313}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:11,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f60583096029c9557b712df1333409e7: 2023-02-28 04:57:11,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7., pid=72, masterSystemTime=1677560231249 2023-02-28 04:57:11,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:11,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:11,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:11,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => bd66d5b12692aefad5974a6eb15a4f47, NAME => 'testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.', STARTKEY => '', ENDKEY => '1'} 2023-02-28 04:57:11,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:11,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:11,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:11,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:11,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:11,285 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:11,285 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560231285"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560231285"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560231285"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560231285"}]},"ts":"1677560231285"} 2023-02-28 04:57:11,286 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e88d2578d5279f76f70975f8fd2b4155; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62452639, jitterRate=-0.0693831592798233}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:11,286 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e88d2578d5279f76f70975f8fd2b4155: 2023-02-28 04:57:11,286 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:11,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:11,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:11,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:11,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 17980904c451ad53a64d5259eea8aec2, NAME => 'testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.', STARTKEY => '4', ENDKEY => '5'} 2023-02-28 04:57:11,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:11,287 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:11,288 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155., pid=75, masterSystemTime=1677560231251 2023-02-28 04:57:11,288 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560231287"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560231287"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560231287"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560231287"}]},"ts":"1677560231287"} 2023-02-28 04:57:11,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:11,288 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:11,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:11,289 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf 2023-02-28 04:57:11,289 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf 2023-02-28 04:57:11,290 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bd66d5b12692aefad5974a6eb15a4f47 columnFamilyName cf 2023-02-28 04:57:11,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:11,290 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:11,291 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:11,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 554a5845ea85f8afefc80a4e37e38fdc, NAME => 'testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.', STARTKEY => '2', ENDKEY => '3'} 2023-02-28 04:57:11,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:11,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:11,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:11,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:11,292 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:11,292 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:11,292 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560231292"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560231292"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560231292"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560231292"}]},"ts":"1677560231292"} 2023-02-28 04:57:11,293 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:11,294 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf 2023-02-28 04:57:11,294 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf 2023-02-28 04:57:11,294 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 17980904c451ad53a64d5259eea8aec2 columnFamilyName cf 2023-02-28 04:57:11,294 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=67 2023-02-28 04:57:11,295 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=67, state=SUCCESS; OpenRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,38247,1677560224887 in 194 msec 2023-02-28 04:57:11,295 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf 2023-02-28 04:57:11,295 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf 2023-02-28 04:57:11,296 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 554a5845ea85f8afefc80a4e37e38fdc columnFamilyName cf 2023-02-28 04:57:11,297 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=65 2023-02-28 04:57:11,297 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=65, state=SUCCESS; OpenRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,46629,1677560224813 in 195 msec 2023-02-28 04:57:11,297 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN in 356 msec 2023-02-28 04:57:11,299 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN in 359 msec 2023-02-28 04:57:11,301 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=75, resume processing ppid=69 2023-02-28 04:57:11,301 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=69, state=SUCCESS; OpenRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,36597,1677560224857 in 193 msec 2023-02-28 04:57:11,302 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/7c2abaa2c369456bb51f5b5e910c2efc 2023-02-28 04:57:11,303 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=69, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN in 363 msec 2023-02-28 04:57:11,303 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(310): Store=bd66d5b12692aefad5974a6eb15a4f47/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:11,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:11,306 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:11,310 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/ffded11160e44993a6ff131b4e9f7276 2023-02-28 04:57:11,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:11,310 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(310): Store=554a5845ea85f8afefc80a4e37e38fdc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:11,311 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened bd66d5b12692aefad5974a6eb15a4f47; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69701915, jitterRate=0.03863947093486786}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:11,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for bd66d5b12692aefad5974a6eb15a4f47: 2023-02-28 04:57:11,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:11,312 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47., pid=71, masterSystemTime=1677560231246 2023-02-28 04:57:11,313 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:11,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:11,314 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:11,315 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:11,315 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560231315"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560231315"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560231315"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560231315"}]},"ts":"1677560231315"} 2023-02-28 04:57:11,316 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/f336080ab5da417c9ad6f84197eeb6c5 2023-02-28 04:57:11,316 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(310): Store=17980904c451ad53a64d5259eea8aec2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:11,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:11,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:11,319 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 554a5845ea85f8afefc80a4e37e38fdc; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68455896, jitterRate=0.020072340965270996}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:11,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 554a5845ea85f8afefc80a4e37e38fdc: 2023-02-28 04:57:11,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:11,320 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=64 2023-02-28 04:57:11,320 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=64, state=SUCCESS; OpenRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,38247,1677560224887 in 223 msec 2023-02-28 04:57:11,321 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc., pid=73, masterSystemTime=1677560231251 2023-02-28 04:57:11,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:11,324 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:11,324 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN in 383 msec 2023-02-28 04:57:11,325 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:11,325 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560231325"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560231325"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560231325"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560231325"}]},"ts":"1677560231325"} 2023-02-28 04:57:11,326 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:11,327 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 17980904c451ad53a64d5259eea8aec2; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69265862, jitterRate=0.03214177489280701}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:11,327 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 17980904c451ad53a64d5259eea8aec2: 2023-02-28 04:57:11,328 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2., pid=74, masterSystemTime=1677560231249 2023-02-28 04:57:11,330 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:11,330 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:11,330 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=66 2023-02-28 04:57:11,330 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=66, state=SUCCESS; OpenRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,36597,1677560224857 in 230 msec 2023-02-28 04:57:11,330 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:11,331 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560231330"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560231330"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560231330"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560231330"}]},"ts":"1677560231330"} 2023-02-28 04:57:11,332 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN in 392 msec 2023-02-28 04:57:11,335 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=68 2023-02-28 04:57:11,335 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=68, state=SUCCESS; OpenRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,46629,1677560224813 in 235 msec 2023-02-28 04:57:11,337 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=68, resume processing ppid=63 2023-02-28 04:57:11,337 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=68, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN in 397 msec 2023-02-28 04:57:11,338 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560231338"}]},"ts":"1677560231338"} 2023-02-28 04:57:11,339 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=ENABLED in hbase:meta 2023-02-28 04:57:11,341 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1677560227585' was successfully enabled. 2023-02-28 04:57:11,343 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1677560227585 in 417 msec 2023-02-28 04:57:11,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-28 04:57:11,681 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1677560227585, procId: 63 completed 2023-02-28 04:57:11,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:11,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:11,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:11,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:11,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:11,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:11,705 DEBUG [Listener at localhost/36897] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1677560227585 2023-02-28 04:57:11,705 DEBUG [Listener at localhost/36897] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:11,707 INFO [Listener at localhost/36897] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-28 04:57:11,974 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-28 04:57:12,073 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-28 04:57:12,125 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-28 04:57:12,149 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-28 04:57:12,150 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-28 04:57:12,305 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-28 04:57:12,470 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-28 04:57:12,589 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-28 04:57:13,822 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-28 04:57:13,822 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-28 04:57:13,985 DEBUG [Listener at localhost/36897] zookeeper.ReadOnlyZKClient(139): Connect 0x20ceba7a to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:13,991 DEBUG [Listener at localhost/36897] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14908088, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:13,993 DEBUG [hconnection-0x40c691bd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:13,996 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59584, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:14,129 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-28 04:57:14,133 DEBUG [hconnection-0x40c691bd-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:14,136 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44908, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:14,239 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-28 04:57:14,242 DEBUG [hconnection-0x40c691bd-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:14,245 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37254, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:14,368 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-28 04:57:14,500 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-28 04:57:14,594 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-28 04:57:15,725 INFO [Listener at localhost/36897] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1677560227585 2023-02-28 04:57:15,725 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1677560227585 2023-02-28 04:57:15,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=76, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:15,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-28 04:57:15,730 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560235729"}]},"ts":"1677560235729"} 2023-02-28 04:57:15,731 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=DISABLING in hbase:meta 2023-02-28 04:57:15,734 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1677560227585 to state=DISABLING 2023-02-28 04:57:15,734 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN}, {pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN}, {pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN}, {pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN}, {pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN}, {pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN}] 2023-02-28 04:57:15,737 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN 2023-02-28 04:57:15,737 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN 2023-02-28 04:57:15,737 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN 2023-02-28 04:57:15,737 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN 2023-02-28 04:57:15,738 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN 2023-02-28 04:57:15,738 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:15,738 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560235738"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560235738"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560235738"}]},"ts":"1677560235738"} 2023-02-28 04:57:15,738 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:15,738 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:15,738 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:15,738 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560235738"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560235738"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560235738"}]},"ts":"1677560235738"} 2023-02-28 04:57:15,738 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560235738"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560235738"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560235738"}]},"ts":"1677560235738"} 2023-02-28 04:57:15,738 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:15,739 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560235738"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560235738"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560235738"}]},"ts":"1677560235738"} 2023-02-28 04:57:15,739 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560235738"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560235738"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560235738"}]},"ts":"1677560235738"} 2023-02-28 04:57:15,740 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=78, state=RUNNABLE; CloseRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:15,741 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=84, ppid=77, state=RUNNABLE; CloseRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:15,742 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=85, ppid=80, state=RUNNABLE; CloseRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:15,743 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=82, state=RUNNABLE; CloseRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:15,745 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=79, state=RUNNABLE; CloseRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:15,745 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN 2023-02-28 04:57:15,746 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:15,746 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560235746"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560235746"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560235746"}]},"ts":"1677560235746"} 2023-02-28 04:57:15,748 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=88, ppid=81, state=RUNNABLE; CloseRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:15,893 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:15,894 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 17980904c451ad53a64d5259eea8aec2, disabling compactions & flushes 2023-02-28 04:57:15,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:15,894 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:15,894 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. after waiting 0 ms 2023-02-28 04:57:15,894 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:15,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 17980904c451ad53a64d5259eea8aec2 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-28 04:57:15,895 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:15,895 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing bd66d5b12692aefad5974a6eb15a4f47, disabling compactions & flushes 2023-02-28 04:57:15,895 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:15,895 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:15,895 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. after waiting 0 ms 2023-02-28 04:57:15,895 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:15,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing bd66d5b12692aefad5974a6eb15a4f47 1/1 column families, dataSize=2.68 KB heapSize=6.02 KB 2023-02-28 04:57:15,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:15,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 554a5845ea85f8afefc80a4e37e38fdc, disabling compactions & flushes 2023-02-28 04:57:15,897 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:15,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:15,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. after waiting 0 ms 2023-02-28 04:57:15,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:15,898 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 554a5845ea85f8afefc80a4e37e38fdc 1/1 column families, dataSize=1.44 KB heapSize=3.34 KB 2023-02-28 04:57:15,933 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 2023-02-28 04:57:15,933 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 2023-02-28 04:57:15,934 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e 2023-02-28 04:57:15,944 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/.tmp/cf/0d9ff88f20374c829118b8980be01f63 2023-02-28 04:57:15,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/.tmp/cf/70a66588039b48cf91a99eb0ba0b59af 2023-02-28 04:57:15,946 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/.tmp/cf/aa2a6ee6b78142d78fa00363c7ea7a6e 2023-02-28 04:57:15,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/.tmp/cf/70a66588039b48cf91a99eb0ba0b59af as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/70a66588039b48cf91a99eb0ba0b59af 2023-02-28 04:57:15,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/.tmp/cf/aa2a6ee6b78142d78fa00363c7ea7a6e as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/aa2a6ee6b78142d78fa00363c7ea7a6e 2023-02-28 04:57:15,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/.tmp/cf/0d9ff88f20374c829118b8980be01f63 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/0d9ff88f20374c829118b8980be01f63 2023-02-28 04:57:15,953 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/70a66588039b48cf91a99eb0ba0b59af, entries=41, sequenceid=15, filesize=11.9 K 2023-02-28 04:57:15,953 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/0d9ff88f20374c829118b8980be01f63, entries=31, sequenceid=15, filesize=10.2 K 2023-02-28 04:57:15,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/aa2a6ee6b78142d78fa00363c7ea7a6e, entries=22, sequenceid=15, filesize=8.6 K 2023-02-28 04:57:15,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.68 KB/2745, heapSize ~6 KB/6144, currentSize=0 B/0 for bd66d5b12692aefad5974a6eb15a4f47 in 59ms, sequenceid=15, compaction requested=false 2023-02-28 04:57:15,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 17980904c451ad53a64d5259eea8aec2 in 60ms, sequenceid=15, compaction requested=false 2023-02-28 04:57:15,955 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.44 KB/1472, heapSize ~3.33 KB/3408, currentSize=0 B/0 for 554a5845ea85f8afefc80a4e37e38fdc in 58ms, sequenceid=15, compaction requested=false 2023-02-28 04:57:15,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-28 04:57:15,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-28 04:57:15,965 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-28 04:57:15,966 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:15,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 17980904c451ad53a64d5259eea8aec2: 2023-02-28 04:57:15,967 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:15,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 554a5845ea85f8afefc80a4e37e38fdc: 2023-02-28 04:57:15,968 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:15,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for bd66d5b12692aefad5974a6eb15a4f47: 2023-02-28 04:57:15,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:15,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f60583096029c9557b712df1333409e7 2023-02-28 04:57:15,970 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f60583096029c9557b712df1333409e7, disabling compactions & flushes 2023-02-28 04:57:15,970 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:15,970 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:15,970 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. after waiting 0 ms 2023-02-28 04:57:15,970 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:15,970 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing f60583096029c9557b712df1333409e7 1/1 column families, dataSize=2.75 KB heapSize=6.16 KB 2023-02-28 04:57:15,971 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=CLOSED 2023-02-28 04:57:15,971 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560235970"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560235970"}]},"ts":"1677560235970"} 2023-02-28 04:57:15,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:15,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:15,972 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e88d2578d5279f76f70975f8fd2b4155, disabling compactions & flushes 2023-02-28 04:57:15,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:15,972 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:15,972 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. after waiting 0 ms 2023-02-28 04:57:15,972 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:15,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing e88d2578d5279f76f70975f8fd2b4155 1/1 column families, dataSize=21.98 KB heapSize=47.50 KB 2023-02-28 04:57:15,972 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=CLOSED 2023-02-28 04:57:15,973 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560235972"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560235972"}]},"ts":"1677560235972"} 2023-02-28 04:57:15,973 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:15,973 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:15,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 815cdd9dbbb8a2be3e8e0c8e96dd83ee, disabling compactions & flushes 2023-02-28 04:57:15,973 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:15,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:15,973 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. after waiting 0 ms 2023-02-28 04:57:15,974 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:15,974 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=CLOSED 2023-02-28 04:57:15,974 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 815cdd9dbbb8a2be3e8e0c8e96dd83ee 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-28 04:57:15,974 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560235974"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560235974"}]},"ts":"1677560235974"} 2023-02-28 04:57:15,980 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=88, resume processing ppid=81 2023-02-28 04:57:15,980 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=88, ppid=81, state=SUCCESS; CloseRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,46629,1677560224813 in 227 msec 2023-02-28 04:57:15,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-28 04:57:15,986 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=79 2023-02-28 04:57:15,986 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=79, state=SUCCESS; CloseRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,36597,1677560224857 in 232 msec 2023-02-28 04:57:15,990 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN in 246 msec 2023-02-28 04:57:15,991 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=84, resume processing ppid=77 2023-02-28 04:57:15,991 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=84, ppid=77, state=SUCCESS; CloseRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,38247,1677560224887 in 238 msec 2023-02-28 04:57:15,992 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=79, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN in 252 msec 2023-02-28 04:57:15,992 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN in 257 msec 2023-02-28 04:57:16,013 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 2023-02-28 04:57:16,016 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f 2023-02-28 04:57:16,018 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 2023-02-28 04:57:16,037 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/.tmp/cf/94af33cb64e64572bdc29544552b5ca3 2023-02-28 04:57:16,042 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/.tmp/cf/1244308735a24059b25d7ce84af068c6 2023-02-28 04:57:16,043 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/.tmp/cf/94af33cb64e64572bdc29544552b5ca3 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/94af33cb64e64572bdc29544552b5ca3 2023-02-28 04:57:16,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/.tmp/cf/1244308735a24059b25d7ce84af068c6 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/1244308735a24059b25d7ce84af068c6 2023-02-28 04:57:16,048 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/94af33cb64e64572bdc29544552b5ca3, entries=28, sequenceid=15, filesize=9.7 K 2023-02-28 04:57:16,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for 815cdd9dbbb8a2be3e8e0c8e96dd83ee in 76ms, sequenceid=15, compaction requested=false 2023-02-28 04:57:16,054 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=22.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/.tmp/cf/9eebf518fd074c118476c5c366e7cdf1 2023-02-28 04:57:16,057 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/1244308735a24059b25d7ce84af068c6, entries=42, sequenceid=15, filesize=12.0 K 2023-02-28 04:57:16,059 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-28 04:57:16,059 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.75 KB/2812, heapSize ~6.14 KB/6288, currentSize=0 B/0 for f60583096029c9557b712df1333409e7 in 89ms, sequenceid=15, compaction requested=false 2023-02-28 04:57:16,064 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/.tmp/cf/9eebf518fd074c118476c5c366e7cdf1 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/9eebf518fd074c118476c5c366e7cdf1 2023-02-28 04:57:16,064 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:16,064 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 815cdd9dbbb8a2be3e8e0c8e96dd83ee: 2023-02-28 04:57:16,066 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:16,067 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=CLOSED 2023-02-28 04:57:16,067 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560236067"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560236067"}]},"ts":"1677560236067"} 2023-02-28 04:57:16,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-28 04:57:16,070 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/9eebf518fd074c118476c5c366e7cdf1, entries=336, sequenceid=15, filesize=62.2 K 2023-02-28 04:57:16,071 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:16,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f60583096029c9557b712df1333409e7: 2023-02-28 04:57:16,071 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.98 KB/22508, heapSize ~47.48 KB/48624, currentSize=0 B/0 for e88d2578d5279f76f70975f8fd2b4155 in 99ms, sequenceid=15, compaction requested=false 2023-02-28 04:57:16,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f60583096029c9557b712df1333409e7 2023-02-28 04:57:16,073 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=85, resume processing ppid=80 2023-02-28 04:57:16,073 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=85, ppid=80, state=SUCCESS; CloseRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,38247,1677560224887 in 327 msec 2023-02-28 04:57:16,073 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=CLOSED 2023-02-28 04:57:16,073 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560236073"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560236073"}]},"ts":"1677560236073"} 2023-02-28 04:57:16,077 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN in 339 msec 2023-02-28 04:57:16,080 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=78 2023-02-28 04:57:16,080 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=78, state=SUCCESS; CloseRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,46629,1677560224813 in 338 msec 2023-02-28 04:57:16,081 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN in 346 msec 2023-02-28 04:57:16,084 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-28 04:57:16,086 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:16,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e88d2578d5279f76f70975f8fd2b4155: 2023-02-28 04:57:16,087 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:16,088 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=CLOSED 2023-02-28 04:57:16,088 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560236088"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560236088"}]},"ts":"1677560236088"} 2023-02-28 04:57:16,091 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=82 2023-02-28 04:57:16,092 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=82, state=SUCCESS; CloseRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,36597,1677560224857 in 347 msec 2023-02-28 04:57:16,094 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=82, resume processing ppid=76 2023-02-28 04:57:16,094 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN in 358 msec 2023-02-28 04:57:16,094 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560236094"}]},"ts":"1677560236094"} 2023-02-28 04:57:16,096 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=DISABLED in hbase:meta 2023-02-28 04:57:16,099 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1677560227585 to state=DISABLED 2023-02-28 04:57:16,101 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=76, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1677560227585 in 375 msec 2023-02-28 04:57:16,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-28 04:57:16,483 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1677560227585, procId: 76 completed 2023-02-28 04:57:16,484 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb1-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:16,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1677560236484 (current time:1677560236484). 2023-02-28 04:57:16,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-28 04:57:16,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] zookeeper.ReadOnlyZKClient(139): Connect 0x32a9c3ea to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:16,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@144bfdfb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:16,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:16,501 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59598, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:16,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32a9c3ea to 127.0.0.1:54355 2023-02-28 04:57:16,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:16,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-28 04:57:16,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb1-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:16,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:16,507 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:16,508 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=89, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE; timeout=600000, timestamp=1677560836508 2023-02-28 04:57:16,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb1-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:16,508 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb1-1677560227585 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1677560227585 2023-02-28 04:57:16,509 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb1-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-28 04:57:16,509 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-28 04:57:16,543 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1677560227585 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } 2023-02-28 04:57:16,548 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:16,549 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:16,549 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:16,549 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:16,549 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:16,549 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:16,549 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:16,550 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:16,549 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:16,550 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:16,550 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:16,549 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:16,550 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:16,551 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:16,553 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/1244308735a24059b25d7ce84af068c6, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/15b46826798648acb28930b8ad9e7857] hfiles 2023-02-28 04:57:16,553 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/1244308735a24059b25d7ce84af068c6 2023-02-28 04:57:16,553 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/15b46826798648acb28930b8ad9e7857 2023-02-28 04:57:16,554 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/0d9ff88f20374c829118b8980be01f63, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/f336080ab5da417c9ad6f84197eeb6c5] hfiles 2023-02-28 04:57:16,554 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/0d9ff88f20374c829118b8980be01f63 2023-02-28 04:57:16,554 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/f336080ab5da417c9ad6f84197eeb6c5 2023-02-28 04:57:16,554 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/70a66588039b48cf91a99eb0ba0b59af, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/7c2abaa2c369456bb51f5b5e910c2efc] hfiles 2023-02-28 04:57:16,554 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/70a66588039b48cf91a99eb0ba0b59af 2023-02-28 04:57:16,555 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/7c2abaa2c369456bb51f5b5e910c2efc 2023-02-28 04:57:16,555 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/94af33cb64e64572bdc29544552b5ca3, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/b4ee3e4950594313bbb87aa56d665751] hfiles 2023-02-28 04:57:16,555 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/94af33cb64e64572bdc29544552b5ca3 2023-02-28 04:57:16,555 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/b4ee3e4950594313bbb87aa56d665751 2023-02-28 04:57:16,555 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/aa2a6ee6b78142d78fa00363c7ea7a6e, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/ffded11160e44993a6ff131b4e9f7276] hfiles 2023-02-28 04:57:16,555 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/aa2a6ee6b78142d78fa00363c7ea7a6e 2023-02-28 04:57:16,555 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/ffded11160e44993a6ff131b4e9f7276 2023-02-28 04:57:16,556 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/7320fea4678244c4b9b33f44fe5a0756, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/9eebf518fd074c118476c5c366e7cdf1] hfiles 2023-02-28 04:57:16,556 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/7320fea4678244c4b9b33f44fe5a0756 2023-02-28 04:57:16,556 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/9eebf518fd074c118476c5c366e7cdf1 2023-02-28 04:57:16,560 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f] hfiles 2023-02-28 04:57:16,560 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 2023-02-28 04:57:16,560 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 2023-02-28 04:57:16,560 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 2023-02-28 04:57:16,560 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 2023-02-28 04:57:16,560 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 2023-02-28 04:57:16,560 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 2023-02-28 04:57:16,560 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b 2023-02-28 04:57:16,560 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e 2023-02-28 04:57:16,560 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 2023-02-28 04:57:16,560 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 2023-02-28 04:57:16,560 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 2023-02-28 04:57:16,561 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f 2023-02-28 04:57:16,759 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-28 04:57:16,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } is done 2023-02-28 04:57:16,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(404): Snapshoting '{ ss=snaptb1-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 }' is still in progress! 2023-02-28 04:57:16,761 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-28 04:57:17,019 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb1-1677560227585 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } as finished. 2023-02-28 04:57:17,019 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb1-1677560227585 2023-02-28 04:57:17,020 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/snaptb1-1677560227585 2023-02-28 04:57:17,049 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/snaptb1-1677560227585 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/snaptb1-1677560227585 2023-02-28 04:57:17,072 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb1-1677560227585 of table testRestoreSnapshot-1677560227585 completed 2023-02-28 04:57:17,072 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/snaptb1-1677560227585 2023-02-28 04:57:17,073 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/snaptb1-1677560227585 2023-02-28 04:57:17,073 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb1-1677560227585 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1677560227585 at 1677560236509Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1677560227585 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } at 1677560236543 (+34 ms)Consolidate snapshot: snaptb1-1677560227585 at 1677560237019 (+476 ms)Loading Region manifests for snaptb1-1677560227585 at 1677560237019Writing data manifest for snaptb1-1677560227585 at 1677560237023 (+4 ms)Verifying snapshot: snaptb1-1677560227585 at 1677560237035 (+12 ms)Snapshot snaptb1-1677560227585 of table testRestoreSnapshot-1677560227585 completed at 1677560237072 (+37 ms) 2023-02-28 04:57:17,074 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=89, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:17,075 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=89, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE in 569 msec 2023-02-28 04:57:17,261 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-28 04:57:17,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } is done 2023-02-28 04:57:17,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb1-1677560227585 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-28 04:57:17,277 INFO [Listener at localhost/36897] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb0-1677560227585-1677560237277 2023-02-28 04:57:17,279 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb0-1677560227585-1677560237277 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:17,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1677560237279 (current time:1677560237279). 2023-02-28 04:57:17,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-28 04:57:17,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] zookeeper.ReadOnlyZKClient(139): Connect 0x4df8ee28 to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:17,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e7707a5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:17,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:17,288 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59608, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:17,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4df8ee28 to 127.0.0.1:54355 2023-02-28 04:57:17,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:17,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-28 04:57:17,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb0-1677560227585-1677560237277 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:17,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:17,293 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:17,294 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=90, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE; timeout=600000, timestamp=1677560837294 2023-02-28 04:57:17,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb0-1677560227585-1677560237277 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:17,294 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb0-1677560227585-1677560237277 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1677560227585 2023-02-28 04:57:17,295 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb0-1677560227585-1677560237277 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-28 04:57:17,295 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-28 04:57:17,309 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1677560227585-1677560237277 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } 2023-02-28 04:57:17,313 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:17,313 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:17,313 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:17,313 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:17,313 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:17,313 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:17,313 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:17,313 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:17,313 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:17,313 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:17,313 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:17,313 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:17,313 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:17,314 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:17,316 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/0d9ff88f20374c829118b8980be01f63, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/f336080ab5da417c9ad6f84197eeb6c5] hfiles 2023-02-28 04:57:17,316 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/0d9ff88f20374c829118b8980be01f63 2023-02-28 04:57:17,316 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/f336080ab5da417c9ad6f84197eeb6c5 2023-02-28 04:57:17,317 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/aa2a6ee6b78142d78fa00363c7ea7a6e, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/ffded11160e44993a6ff131b4e9f7276] hfiles 2023-02-28 04:57:17,317 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/aa2a6ee6b78142d78fa00363c7ea7a6e 2023-02-28 04:57:17,317 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/ffded11160e44993a6ff131b4e9f7276 2023-02-28 04:57:17,318 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/70a66588039b48cf91a99eb0ba0b59af, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/7c2abaa2c369456bb51f5b5e910c2efc] hfiles 2023-02-28 04:57:17,318 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/7320fea4678244c4b9b33f44fe5a0756, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/9eebf518fd074c118476c5c366e7cdf1] hfiles 2023-02-28 04:57:17,318 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/70a66588039b48cf91a99eb0ba0b59af 2023-02-28 04:57:17,318 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/94af33cb64e64572bdc29544552b5ca3, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/b4ee3e4950594313bbb87aa56d665751] hfiles 2023-02-28 04:57:17,318 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/7320fea4678244c4b9b33f44fe5a0756 2023-02-28 04:57:17,318 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/94af33cb64e64572bdc29544552b5ca3 2023-02-28 04:57:17,318 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/7c2abaa2c369456bb51f5b5e910c2efc 2023-02-28 04:57:17,318 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/1244308735a24059b25d7ce84af068c6, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/15b46826798648acb28930b8ad9e7857] hfiles 2023-02-28 04:57:17,318 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/1244308735a24059b25d7ce84af068c6 2023-02-28 04:57:17,318 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/b4ee3e4950594313bbb87aa56d665751 2023-02-28 04:57:17,318 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/9eebf518fd074c118476c5c366e7cdf1 2023-02-28 04:57:17,318 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/15b46826798648acb28930b8ad9e7857 2023-02-28 04:57:17,326 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f] hfiles 2023-02-28 04:57:17,326 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 2023-02-28 04:57:17,326 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 2023-02-28 04:57:17,326 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 2023-02-28 04:57:17,326 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 2023-02-28 04:57:17,326 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 2023-02-28 04:57:17,326 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 2023-02-28 04:57:17,326 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b 2023-02-28 04:57:17,326 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e 2023-02-28 04:57:17,327 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 2023-02-28 04:57:17,327 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 2023-02-28 04:57:17,327 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 2023-02-28 04:57:17,327 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f 2023-02-28 04:57:17,359 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb0-1677560227585-1677560237277 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } as finished. 2023-02-28 04:57:17,359 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb0-1677560227585-1677560237277 2023-02-28 04:57:17,360 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1677560227585-1677560237277 2023-02-28 04:57:17,408 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20ceba7a to 127.0.0.1:54355 2023-02-28 04:57:17,408 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:17,411 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1677560227585-1677560237277 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/hbase-failsafe-snaptb0-1677560227585-1677560237277 2023-02-28 04:57:17,436 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb0-1677560227585-1677560237277 of table testRestoreSnapshot-1677560227585 completed 2023-02-28 04:57:17,437 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1677560227585-1677560237277 2023-02-28 04:57:17,437 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1677560227585-1677560237277 2023-02-28 04:57:17,437 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb0-1677560227585-1677560237277 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1677560227585 at 1677560237294Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1677560227585-1677560237277 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } at 1677560237309 (+15 ms)Consolidate snapshot: hbase-failsafe-snaptb0-1677560227585-1677560237277 at 1677560237359 (+50 ms)Loading Region manifests for hbase-failsafe-snaptb0-1677560227585-1677560237277 at 1677560237359Writing data manifest for hbase-failsafe-snaptb0-1677560227585-1677560237277 at 1677560237364 (+5 ms)Verifying snapshot: hbase-failsafe-snaptb0-1677560227585-1677560237277 at 1677560237373 (+9 ms)Snapshot hbase-failsafe-snaptb0-1677560227585-1677560237277 of table testRestoreSnapshot-1677560227585 completed at 1677560237436 (+63 ms) 2023-02-28 04:57:17,438 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=90, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:17,440 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=90, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE in 147 msec 2023-02-28 04:57:17,545 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-28 04:57:17,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1677560227585-1677560237277 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } is done 2023-02-28 04:57:17,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb0-1677560227585-1677560237277 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-28 04:57:17,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=91, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1677560227585 snapshot=name: "snaptb0-1677560227585" table: "testRestoreSnapshot-1677560227585" creation_time: 1677560230656 type: DISABLED version: 2 ttl: 0 ) 2023-02-28 04:57:17,571 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(880): Restore snapshot=snaptb0-1677560227585 as table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:17,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-28 04:57:17,586 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/.tabledesc/.tableinfo.0000000002 2023-02-28 04:57:17,587 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/.tabledesc/.tableinfo.0000000001 2023-02-28 04:57:17,587 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/.tabledesc/.tableinfo.0000000002 2023-02-28 04:57:17,588 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb0-1677560227585 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } 2023-02-28 04:57:17,591 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb0-1677560227585" table: "testRestoreSnapshot-1677560227585" creation_time: 1677560230656 type: DISABLED version: 2 ttl: 0 2023-02-28 04:57:17,592 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585 2023-02-28 04:57:17,600 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:17,606 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 2023-02-28 04:57:17,608 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 2023-02-28 04:57:17,610 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f 2023-02-28 04:57:17,611 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e 2023-02-28 04:57:17,612 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 2023-02-28 04:57:17,614 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 2023-02-28 04:57:17,614 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:17,614 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:17,614 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:17,614 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:17,614 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:17,614 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(223): region to restore: f60583096029c9557b712df1333409e7 2023-02-28 04:57:17,624 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/0d9ff88f20374c829118b8980be01f63 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/0d9ff88f20374c829118b8980be01f63 2023-02-28 04:57:17,625 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/9eebf518fd074c118476c5c366e7cdf1 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/9eebf518fd074c118476c5c366e7cdf1 2023-02-28 04:57:17,626 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/70a66588039b48cf91a99eb0ba0b59af to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/70a66588039b48cf91a99eb0ba0b59af 2023-02-28 04:57:17,626 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/aa2a6ee6b78142d78fa00363c7ea7a6e to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/aa2a6ee6b78142d78fa00363c7ea7a6e 2023-02-28 04:57:17,626 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/94af33cb64e64572bdc29544552b5ca3 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/94af33cb64e64572bdc29544552b5ca3 2023-02-28 04:57:17,626 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/1244308735a24059b25d7ce84af068c6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/1244308735a24059b25d7ce84af068c6 2023-02-28 04:57:17,627 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb0-1677560227585" table: "testRestoreSnapshot-1677560227585" creation_time: 1677560230656 type: DISABLED version: 2 ttl: 0 2023-02-28 04:57:17,628 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560237628"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:17,628 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560237628"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:17,629 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560237628"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:17,629 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560237628"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:17,629 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560237628"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:17,629 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560237628"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:17,632 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-28 04:57:17,632 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 17980904c451ad53a64d5259eea8aec2, NAME => 'testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 554a5845ea85f8afefc80a4e37e38fdc, NAME => 'testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 815cdd9dbbb8a2be3e8e0c8e96dd83ee, NAME => 'testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => bd66d5b12692aefad5974a6eb15a4f47, NAME => 'testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => e88d2578d5279f76f70975f8fd2b4155, NAME => 'testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => f60583096029c9557b712df1333409e7, NAME => 'testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-28 04:57:17,632 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560237629"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560237629"}]},"ts":"1677560237629"} 2023-02-28 04:57:17,632 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560237629"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560237629"}]},"ts":"1677560237629"} 2023-02-28 04:57:17,632 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560237629"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560237629"}]},"ts":"1677560237629"} 2023-02-28 04:57:17,632 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560237629"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560237629"}]},"ts":"1677560237629"} 2023-02-28 04:57:17,632 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560237629"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560237629"}]},"ts":"1677560237629"} 2023-02-28 04:57:17,633 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560237629"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560237629"}]},"ts":"1677560237629"} 2023-02-28 04:57:17,635 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-28 04:57:17,635 INFO [PEWorker-1] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-28 04:57:17,635 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 17980904c451ad53a64d5259eea8aec2, NAME => 'testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 554a5845ea85f8afefc80a4e37e38fdc, NAME => 'testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 815cdd9dbbb8a2be3e8e0c8e96dd83ee, NAME => 'testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => bd66d5b12692aefad5974a6eb15a4f47, NAME => 'testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => e88d2578d5279f76f70975f8fd2b4155, NAME => 'testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => f60583096029c9557b712df1333409e7, NAME => 'testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-28 04:57:17,636 INFO [PEWorker-1] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb0-1677560227585 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1677560227585 completed! 2023-02-28 04:57:17,638 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=91, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1677560227585 snapshot=name: "snaptb0-1677560227585" table: "testRestoreSnapshot-1677560227585" creation_time: 1677560230656 type: DISABLED version: 2 ttl: 0 ) in 74 msec 2023-02-28 04:57:17,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-28 04:57:17,827 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1677560227585, procId: 91 completed 2023-02-28 04:57:17,827 INFO [Listener at localhost/36897] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb0-1677560227585-1677560237277 2023-02-28 04:57:17,831 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb0-1677560227585-1677560237277" 2023-02-28 04:57:17,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb0-1677560227585-1677560237277 2023-02-28 04:57:17,835 INFO [Listener at localhost/36897] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1677560227585 2023-02-28 04:57:17,835 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1677560227585 2023-02-28 04:57:17,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=92, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:17,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-28 04:57:17,839 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1677560227585 2023-02-28 04:57:17,840 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560237839"}]},"ts":"1677560237839"} 2023-02-28 04:57:17,841 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=ENABLING in hbase:meta 2023-02-28 04:57:17,845 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-28 04:57:17,848 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN}, {pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN}, {pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN}, {pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN}, {pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN}, {pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN}] 2023-02-28 04:57:17,850 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN 2023-02-28 04:57:17,850 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN 2023-02-28 04:57:17,850 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN 2023-02-28 04:57:17,850 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN 2023-02-28 04:57:17,851 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:17,851 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:17,851 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN 2023-02-28 04:57:17,852 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:17,852 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:17,852 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN 2023-02-28 04:57:17,852 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:17,853 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:18,001 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-28 04:57:18,002 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-28 04:57:18,002 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-28 04:57:18,002 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-28 04:57:18,002 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-28 04:57:18,006 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:18,006 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:18,006 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560238006"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560238006"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560238006"}]},"ts":"1677560238006"} 2023-02-28 04:57:18,006 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:18,006 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560238006"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560238006"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560238006"}]},"ts":"1677560238006"} 2023-02-28 04:57:18,006 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:18,006 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:18,007 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560238006"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560238006"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560238006"}]},"ts":"1677560238006"} 2023-02-28 04:57:18,007 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560238006"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560238006"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560238006"}]},"ts":"1677560238006"} 2023-02-28 04:57:18,007 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560238006"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560238006"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560238006"}]},"ts":"1677560238006"} 2023-02-28 04:57:18,009 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=98, state=RUNNABLE; OpenRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:18,010 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=96, state=RUNNABLE; OpenRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:18,011 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=101, ppid=94, state=RUNNABLE; OpenRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:18,012 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=102, ppid=95, state=RUNNABLE; OpenRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:18,013 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:18,013 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=97, state=RUNNABLE; OpenRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:18,013 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560238012"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560238012"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560238012"}]},"ts":"1677560238012"} 2023-02-28 04:57:18,015 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=93, state=RUNNABLE; OpenRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:18,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-28 04:57:18,164 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:18,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 554a5845ea85f8afefc80a4e37e38fdc, NAME => 'testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.', STARTKEY => '2', ENDKEY => '3'} 2023-02-28 04:57:18,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:18,164 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:18,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:18,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 815cdd9dbbb8a2be3e8e0c8e96dd83ee, NAME => 'testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.', STARTKEY => '3', ENDKEY => '4'} 2023-02-28 04:57:18,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:18,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:18,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:18,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:18,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:18,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:18,165 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:18,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f60583096029c9557b712df1333409e7, NAME => 'testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.', STARTKEY => '1', ENDKEY => '2'} 2023-02-28 04:57:18,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 f60583096029c9557b712df1333409e7 2023-02-28 04:57:18,166 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:18,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:18,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f60583096029c9557b712df1333409e7 2023-02-28 04:57:18,166 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f60583096029c9557b712df1333409e7 2023-02-28 04:57:18,166 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:18,167 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf 2023-02-28 04:57:18,167 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf 2023-02-28 04:57:18,167 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf 2023-02-28 04:57:18,167 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf 2023-02-28 04:57:18,167 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f60583096029c9557b712df1333409e7 2023-02-28 04:57:18,168 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 554a5845ea85f8afefc80a4e37e38fdc columnFamilyName cf 2023-02-28 04:57:18,168 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 815cdd9dbbb8a2be3e8e0c8e96dd83ee columnFamilyName cf 2023-02-28 04:57:18,168 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf 2023-02-28 04:57:18,168 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf 2023-02-28 04:57:18,169 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f60583096029c9557b712df1333409e7 columnFamilyName cf 2023-02-28 04:57:18,174 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/b4ee3e4950594313bbb87aa56d665751 2023-02-28 04:57:18,176 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(310): Store=815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:18,177 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:18,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:18,179 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/ffded11160e44993a6ff131b4e9f7276 2023-02-28 04:57:18,179 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/15b46826798648acb28930b8ad9e7857 2023-02-28 04:57:18,179 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(310): Store=554a5845ea85f8afefc80a4e37e38fdc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:18,179 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(310): Store=f60583096029c9557b712df1333409e7/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:18,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:18,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:18,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:18,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:18,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:18,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 815cdd9dbbb8a2be3e8e0c8e96dd83ee; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68010729, jitterRate=0.013438835740089417}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:18,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 815cdd9dbbb8a2be3e8e0c8e96dd83ee: 2023-02-28 04:57:18,183 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee., pid=100, masterSystemTime=1677560238162 2023-02-28 04:57:18,184 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:18,184 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f60583096029c9557b712df1333409e7 2023-02-28 04:57:18,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:18,185 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:18,185 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:18,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => bd66d5b12692aefad5974a6eb15a4f47, NAME => 'testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.', STARTKEY => '', ENDKEY => '1'} 2023-02-28 04:57:18,185 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f60583096029c9557b712df1333409e7; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66275288, jitterRate=-0.012421250343322754}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:18,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:18,185 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 554a5845ea85f8afefc80a4e37e38fdc; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65738433, jitterRate=-0.020421013236045837}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:18,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:18,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 554a5845ea85f8afefc80a4e37e38fdc: 2023-02-28 04:57:18,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:18,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f60583096029c9557b712df1333409e7: 2023-02-28 04:57:18,185 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:18,185 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:18,186 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560238185"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560238185"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560238185"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560238185"}]},"ts":"1677560238185"} 2023-02-28 04:57:18,186 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc., pid=102, masterSystemTime=1677560238161 2023-02-28 04:57:18,186 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7., pid=101, masterSystemTime=1677560238163 2023-02-28 04:57:18,187 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:18,187 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:18,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:18,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:18,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e88d2578d5279f76f70975f8fd2b4155, NAME => 'testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.', STARTKEY => '5', ENDKEY => ''} 2023-02-28 04:57:18,188 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf 2023-02-28 04:57:18,188 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf 2023-02-28 04:57:18,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:18,188 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:18,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:18,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:18,188 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560238188"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560238188"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560238188"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560238188"}]},"ts":"1677560238188"} 2023-02-28 04:57:18,188 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bd66d5b12692aefad5974a6eb15a4f47 columnFamilyName cf 2023-02-28 04:57:18,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:18,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:18,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:18,189 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:18,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 17980904c451ad53a64d5259eea8aec2, NAME => 'testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.', STARTKEY => '4', ENDKEY => '5'} 2023-02-28 04:57:18,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:18,189 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:18,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:18,189 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560238189"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560238189"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560238189"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560238189"}]},"ts":"1677560238189"} 2023-02-28 04:57:18,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:18,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:18,190 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:18,191 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=96 2023-02-28 04:57:18,191 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=96, state=SUCCESS; OpenRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,36597,1677560224857 in 177 msec 2023-02-28 04:57:18,191 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf 2023-02-28 04:57:18,191 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf 2023-02-28 04:57:18,192 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e88d2578d5279f76f70975f8fd2b4155 columnFamilyName cf 2023-02-28 04:57:18,192 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:18,193 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=96, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN in 342 msec 2023-02-28 04:57:18,193 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf 2023-02-28 04:57:18,193 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf 2023-02-28 04:57:18,194 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 17980904c451ad53a64d5259eea8aec2 columnFamilyName cf 2023-02-28 04:57:18,194 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=101, resume processing ppid=94 2023-02-28 04:57:18,194 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=101, ppid=94, state=SUCCESS; OpenRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,38247,1677560224887 in 181 msec 2023-02-28 04:57:18,196 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN in 349 msec 2023-02-28 04:57:18,198 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/f336080ab5da417c9ad6f84197eeb6c5 2023-02-28 04:57:18,199 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(310): Store=17980904c451ad53a64d5259eea8aec2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:18,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:18,200 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/7320fea4678244c4b9b33f44fe5a0756 2023-02-28 04:57:18,200 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(310): Store=e88d2578d5279f76f70975f8fd2b4155/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:18,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:18,202 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=102, resume processing ppid=95 2023-02-28 04:57:18,202 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=102, ppid=95, state=SUCCESS; OpenRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,46629,1677560224813 in 188 msec 2023-02-28 04:57:18,203 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:18,203 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=95, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN in 354 msec 2023-02-28 04:57:18,204 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:18,207 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:18,207 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:18,208 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e88d2578d5279f76f70975f8fd2b4155; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64271848, jitterRate=-0.0422748327255249}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:18,208 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 17980904c451ad53a64d5259eea8aec2; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65415951, jitterRate=-0.025226369500160217}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:18,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e88d2578d5279f76f70975f8fd2b4155: 2023-02-28 04:57:18,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 17980904c451ad53a64d5259eea8aec2: 2023-02-28 04:57:18,209 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/7c2abaa2c369456bb51f5b5e910c2efc 2023-02-28 04:57:18,209 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(310): Store=bd66d5b12692aefad5974a6eb15a4f47/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:18,209 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2., pid=103, masterSystemTime=1677560238163 2023-02-28 04:57:18,209 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155., pid=99, masterSystemTime=1677560238161 2023-02-28 04:57:18,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:18,211 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:18,211 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:18,212 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:18,212 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:18,212 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:18,212 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560238212"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560238212"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560238212"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560238212"}]},"ts":"1677560238212"} 2023-02-28 04:57:18,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:18,213 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:18,213 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560238213"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560238213"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560238213"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560238213"}]},"ts":"1677560238213"} 2023-02-28 04:57:18,216 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:18,217 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened bd66d5b12692aefad5974a6eb15a4f47; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68394989, jitterRate=0.019164755940437317}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:18,217 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for bd66d5b12692aefad5974a6eb15a4f47: 2023-02-28 04:57:18,218 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=97 2023-02-28 04:57:18,218 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=97, state=SUCCESS; OpenRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,38247,1677560224887 in 202 msec 2023-02-28 04:57:18,218 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47., pid=104, masterSystemTime=1677560238162 2023-02-28 04:57:18,219 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=98 2023-02-28 04:57:18,219 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=98, state=SUCCESS; OpenRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,46629,1677560224813 in 207 msec 2023-02-28 04:57:18,222 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN in 370 msec 2023-02-28 04:57:18,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:18,222 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:18,223 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN in 371 msec 2023-02-28 04:57:18,223 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:18,223 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560238223"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560238223"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560238223"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560238223"}]},"ts":"1677560238223"} 2023-02-28 04:57:18,233 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=93 2023-02-28 04:57:18,233 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=93, state=SUCCESS; OpenRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,36597,1677560224857 in 213 msec 2023-02-28 04:57:18,235 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=93, resume processing ppid=92 2023-02-28 04:57:18,236 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN in 388 msec 2023-02-28 04:57:18,236 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560238236"}]},"ts":"1677560238236"} 2023-02-28 04:57:18,237 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=ENABLED in hbase:meta 2023-02-28 04:57:18,239 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1677560227585' was successfully enabled. 2023-02-28 04:57:18,241 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=92, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1677560227585 in 404 msec 2023-02-28 04:57:18,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-28 04:57:18,591 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1677560227585, procId: 92 completed 2023-02-28 04:57:18,591 DEBUG [Listener at localhost/36897] zookeeper.ReadOnlyZKClient(139): Connect 0x3e907963 to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:18,597 DEBUG [Listener at localhost/36897] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@116b3a18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:18,598 DEBUG [hconnection-0x5587b766-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:18,603 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59616, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:18,608 DEBUG [hconnection-0x5587b766-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:18,614 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37260, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:18,722 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-28 04:57:18,820 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-28 04:57:18,824 DEBUG [hconnection-0x5587b766-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:18,828 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44918, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:18,943 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-28 04:57:19,047 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-28 04:57:19,131 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-28 04:57:20,080 INFO [Listener at localhost/36897] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1677560227585 2023-02-28 04:57:20,081 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1677560227585 2023-02-28 04:57:20,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:20,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-28 04:57:20,086 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560240086"}]},"ts":"1677560240086"} 2023-02-28 04:57:20,087 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=DISABLING in hbase:meta 2023-02-28 04:57:20,089 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1677560227585 to state=DISABLING 2023-02-28 04:57:20,090 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN}, {pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN}, {pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN}, {pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN}, {pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN}, {pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN}] 2023-02-28 04:57:20,093 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN 2023-02-28 04:57:20,093 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN 2023-02-28 04:57:20,093 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN 2023-02-28 04:57:20,093 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN 2023-02-28 04:57:20,094 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN 2023-02-28 04:57:20,095 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:20,095 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:20,095 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560240095"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560240095"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560240095"}]},"ts":"1677560240095"} 2023-02-28 04:57:20,095 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560240095"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560240095"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560240095"}]},"ts":"1677560240095"} 2023-02-28 04:57:20,095 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:20,095 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:20,095 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:20,096 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560240095"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560240095"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560240095"}]},"ts":"1677560240095"} 2023-02-28 04:57:20,096 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560240095"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560240095"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560240095"}]},"ts":"1677560240095"} 2023-02-28 04:57:20,096 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560240095"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560240095"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560240095"}]},"ts":"1677560240095"} 2023-02-28 04:57:20,097 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=112, ppid=108, state=RUNNABLE; CloseRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:20,100 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=107, state=RUNNABLE; CloseRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:20,101 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=106, state=RUNNABLE; CloseRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:20,102 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=109, state=RUNNABLE; CloseRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:20,102 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=116, ppid=111, state=RUNNABLE; CloseRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:20,106 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN 2023-02-28 04:57:20,107 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:20,108 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560240107"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560240107"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560240107"}]},"ts":"1677560240107"} 2023-02-28 04:57:20,109 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=110, state=RUNNABLE; CloseRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:20,255 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:20,255 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:20,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 554a5845ea85f8afefc80a4e37e38fdc, disabling compactions & flushes 2023-02-28 04:57:20,256 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:20,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 17980904c451ad53a64d5259eea8aec2, disabling compactions & flushes 2023-02-28 04:57:20,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:20,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:20,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. after waiting 0 ms 2023-02-28 04:57:20,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:20,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:20,259 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. after waiting 0 ms 2023-02-28 04:57:20,259 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:20,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:20,259 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 815cdd9dbbb8a2be3e8e0c8e96dd83ee, disabling compactions & flushes 2023-02-28 04:57:20,259 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:20,259 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:20,259 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. after waiting 0 ms 2023-02-28 04:57:20,259 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:20,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-28 04:57:20,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-28 04:57:20,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-28 04:57:20,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:20,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 554a5845ea85f8afefc80a4e37e38fdc: 2023-02-28 04:57:20,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:20,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:20,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 17980904c451ad53a64d5259eea8aec2: 2023-02-28 04:57:20,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 815cdd9dbbb8a2be3e8e0c8e96dd83ee: 2023-02-28 04:57:20,271 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:20,271 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:20,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e88d2578d5279f76f70975f8fd2b4155, disabling compactions & flushes 2023-02-28 04:57:20,271 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:20,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:20,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. after waiting 0 ms 2023-02-28 04:57:20,271 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:20,271 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=CLOSED 2023-02-28 04:57:20,272 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560240271"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560240271"}]},"ts":"1677560240271"} 2023-02-28 04:57:20,272 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:20,272 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f60583096029c9557b712df1333409e7 2023-02-28 04:57:20,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f60583096029c9557b712df1333409e7, disabling compactions & flushes 2023-02-28 04:57:20,272 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:20,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:20,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. after waiting 0 ms 2023-02-28 04:57:20,272 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:20,273 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=CLOSED 2023-02-28 04:57:20,273 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560240273"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560240273"}]},"ts":"1677560240273"} 2023-02-28 04:57:20,273 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:20,273 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:20,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing bd66d5b12692aefad5974a6eb15a4f47, disabling compactions & flushes 2023-02-28 04:57:20,273 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:20,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:20,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. after waiting 0 ms 2023-02-28 04:57:20,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:20,274 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=CLOSED 2023-02-28 04:57:20,274 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560240274"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560240274"}]},"ts":"1677560240274"} 2023-02-28 04:57:20,275 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-28 04:57:20,276 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:20,276 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e88d2578d5279f76f70975f8fd2b4155: 2023-02-28 04:57:20,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-28 04:57:20,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:20,277 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f60583096029c9557b712df1333409e7: 2023-02-28 04:57:20,278 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-28 04:57:20,278 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:20,278 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for bd66d5b12692aefad5974a6eb15a4f47: 2023-02-28 04:57:20,279 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=112, resume processing ppid=108 2023-02-28 04:57:20,279 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=112, ppid=108, state=SUCCESS; CloseRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,46629,1677560224813 in 177 msec 2023-02-28 04:57:20,279 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:20,279 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=CLOSED 2023-02-28 04:57:20,279 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560240279"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560240279"}]},"ts":"1677560240279"} 2023-02-28 04:57:20,279 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f60583096029c9557b712df1333409e7 2023-02-28 04:57:20,280 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=109 2023-02-28 04:57:20,280 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=109, state=SUCCESS; CloseRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,36597,1677560224857 in 175 msec 2023-02-28 04:57:20,280 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=110 2023-02-28 04:57:20,280 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=110, state=SUCCESS; CloseRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,38247,1677560224887 in 167 msec 2023-02-28 04:57:20,280 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=CLOSED 2023-02-28 04:57:20,281 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560240280"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560240280"}]},"ts":"1677560240280"} 2023-02-28 04:57:20,281 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:20,282 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=108, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN in 189 msec 2023-02-28 04:57:20,283 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=109, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN in 190 msec 2023-02-28 04:57:20,283 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN in 190 msec 2023-02-28 04:57:20,283 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=CLOSED 2023-02-28 04:57:20,283 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560240283"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560240283"}]},"ts":"1677560240283"} 2023-02-28 04:57:20,284 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=111 2023-02-28 04:57:20,284 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=111, state=SUCCESS; CloseRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,46629,1677560224813 in 180 msec 2023-02-28 04:57:20,285 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=107 2023-02-28 04:57:20,285 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=107, state=SUCCESS; CloseRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,38247,1677560224887 in 185 msec 2023-02-28 04:57:20,285 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN in 194 msec 2023-02-28 04:57:20,286 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=106 2023-02-28 04:57:20,286 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=106, state=SUCCESS; CloseRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,36597,1677560224857 in 183 msec 2023-02-28 04:57:20,286 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN in 195 msec 2023-02-28 04:57:20,287 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=106, resume processing ppid=105 2023-02-28 04:57:20,287 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN in 196 msec 2023-02-28 04:57:20,288 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560240288"}]},"ts":"1677560240288"} 2023-02-28 04:57:20,289 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=DISABLED in hbase:meta 2023-02-28 04:57:20,291 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1677560227585 to state=DISABLED 2023-02-28 04:57:20,293 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1677560227585 in 211 msec 2023-02-28 04:57:20,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-28 04:57:20,337 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1677560227585, procId: 105 completed 2023-02-28 04:57:20,346 INFO [Listener at localhost/36897] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1677560227585-1677560240346 2023-02-28 04:57:20,347 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-emptySnaptb-1677560227585-1677560240346 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:20,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1677560240347 (current time:1677560240347). 2023-02-28 04:57:20,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-28 04:57:20,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] zookeeper.ReadOnlyZKClient(139): Connect 0x7fccc5c9 to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:20,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b0e4993, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:20,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:20,358 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59620, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:20,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7fccc5c9 to 127.0.0.1:54355 2023-02-28 04:57:20,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:20,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-28 04:57:20,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-emptySnaptb-1677560227585-1677560240346 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:20,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:20,362 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:20,363 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=118, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE; timeout=600000, timestamp=1677560840363 2023-02-28 04:57:20,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-emptySnaptb-1677560227585-1677560240346 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:20,363 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1677560227585-1677560240346 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1677560227585 2023-02-28 04:57:20,363 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-emptySnaptb-1677560227585-1677560240346 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-28 04:57:20,363 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-28 04:57:20,373 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1677560227585-1677560240346 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } 2023-02-28 04:57:20,375 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:20,375 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:20,375 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:20,375 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:20,375 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:20,375 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:20,375 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:20,375 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:20,376 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:20,375 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:20,376 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:20,375 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:20,375 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:20,376 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:20,377 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/b4ee3e4950594313bbb87aa56d665751] hfiles 2023-02-28 04:57:20,377 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/f336080ab5da417c9ad6f84197eeb6c5] hfiles 2023-02-28 04:57:20,377 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/b4ee3e4950594313bbb87aa56d665751 2023-02-28 04:57:20,378 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/f336080ab5da417c9ad6f84197eeb6c5 2023-02-28 04:57:20,380 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/7320fea4678244c4b9b33f44fe5a0756] hfiles 2023-02-28 04:57:20,380 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/7320fea4678244c4b9b33f44fe5a0756 2023-02-28 04:57:20,380 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/15b46826798648acb28930b8ad9e7857] hfiles 2023-02-28 04:57:20,380 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/15b46826798648acb28930b8ad9e7857 2023-02-28 04:57:20,386 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/ffded11160e44993a6ff131b4e9f7276] hfiles 2023-02-28 04:57:20,386 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/ffded11160e44993a6ff131b4e9f7276 2023-02-28 04:57:20,386 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/7c2abaa2c369456bb51f5b5e910c2efc] hfiles 2023-02-28 04:57:20,386 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/7c2abaa2c369456bb51f5b5e910c2efc 2023-02-28 04:57:20,387 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963] hfiles 2023-02-28 04:57:20,387 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 2023-02-28 04:57:20,388 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 2023-02-28 04:57:20,388 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 2023-02-28 04:57:20,388 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b 2023-02-28 04:57:20,388 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 2023-02-28 04:57:20,388 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 2023-02-28 04:57:20,412 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-emptySnaptb-1677560227585-1677560240346 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } as finished. 2023-02-28 04:57:20,412 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-emptySnaptb-1677560227585-1677560240346 2023-02-28 04:57:20,413 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1677560227585-1677560240346 2023-02-28 04:57:20,433 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1677560227585-1677560240346 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/hbase-failsafe-emptySnaptb-1677560227585-1677560240346 2023-02-28 04:57:20,454 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-emptySnaptb-1677560227585-1677560240346 of table testRestoreSnapshot-1677560227585 completed 2023-02-28 04:57:20,454 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1677560227585-1677560240346 2023-02-28 04:57:20,454 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1677560227585-1677560240346 2023-02-28 04:57:20,454 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1677560227585-1677560240346 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1677560227585 at 1677560240363Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1677560227585-1677560240346 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } at 1677560240373 (+10 ms)Consolidate snapshot: hbase-failsafe-emptySnaptb-1677560227585-1677560240346 at 1677560240412 (+39 ms)Loading Region manifests for hbase-failsafe-emptySnaptb-1677560227585-1677560240346 at 1677560240412Writing data manifest for hbase-failsafe-emptySnaptb-1677560227585-1677560240346 at 1677560240416 (+4 ms)Verifying snapshot: hbase-failsafe-emptySnaptb-1677560227585-1677560240346 at 1677560240426 (+10 ms)Snapshot hbase-failsafe-emptySnaptb-1677560227585-1677560240346 of table testRestoreSnapshot-1677560227585 completed at 1677560240454 (+28 ms) 2023-02-28 04:57:20,456 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=118, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:20,457 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=118, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE in 96 msec 2023-02-28 04:57:20,613 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-28 04:57:20,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1677560227585-1677560240346 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } is done 2023-02-28 04:57:20,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-emptySnaptb-1677560227585-1677560240346 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-28 04:57:20,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1677560227585 snapshot=name: "emptySnaptb-1677560227585" table: "testRestoreSnapshot-1677560227585" creation_time: 1677560228718 type: DISABLED version: 2 ttl: 0 ) 2023-02-28 04:57:20,621 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(880): Restore snapshot=emptySnaptb-1677560227585 as table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:20,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-28 04:57:20,633 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/.tabledesc/.tableinfo.0000000003 2023-02-28 04:57:20,634 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/.tabledesc/.tableinfo.0000000002 2023-02-28 04:57:20,634 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/.tabledesc/.tableinfo.0000000003 2023-02-28 04:57:20,635 INFO [PEWorker-5] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=emptySnaptb-1677560227585 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } 2023-02-28 04:57:20,636 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "emptySnaptb-1677560227585" table: "testRestoreSnapshot-1677560227585" creation_time: 1677560228718 type: DISABLED version: 2 ttl: 0 2023-02-28 04:57:20,636 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585 2023-02-28 04:57:20,643 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:20,646 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 2023-02-28 04:57:20,647 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 2023-02-28 04:57:20,648 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 2023-02-28 04:57:20,650 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b 2023-02-28 04:57:20,651 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 2023-02-28 04:57:20,652 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 2023-02-28 04:57:20,653 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:20,653 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:20,653 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:20,653 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:20,653 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:20,653 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: f60583096029c9557b712df1333409e7 2023-02-28 04:57:20,658 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/f336080ab5da417c9ad6f84197eeb6c5 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/f336080ab5da417c9ad6f84197eeb6c5 2023-02-28 04:57:20,658 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/b4ee3e4950594313bbb87aa56d665751 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/b4ee3e4950594313bbb87aa56d665751 2023-02-28 04:57:20,658 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/7c2abaa2c369456bb51f5b5e910c2efc to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/7c2abaa2c369456bb51f5b5e910c2efc 2023-02-28 04:57:20,660 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/ffded11160e44993a6ff131b4e9f7276 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/ffded11160e44993a6ff131b4e9f7276 2023-02-28 04:57:20,660 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/15b46826798648acb28930b8ad9e7857 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/15b46826798648acb28930b8ad9e7857 2023-02-28 04:57:20,661 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/7320fea4678244c4b9b33f44fe5a0756 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/7320fea4678244c4b9b33f44fe5a0756 2023-02-28 04:57:20,661 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "emptySnaptb-1677560227585" table: "testRestoreSnapshot-1677560227585" creation_time: 1677560228718 type: DISABLED version: 2 ttl: 0 2023-02-28 04:57:20,662 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560240662"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:20,662 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560240662"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:20,663 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560240662"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:20,663 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560240662"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:20,663 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560240662"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:20,663 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560240662"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:20,665 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-28 04:57:20,665 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 17980904c451ad53a64d5259eea8aec2, NAME => 'testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 554a5845ea85f8afefc80a4e37e38fdc, NAME => 'testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 815cdd9dbbb8a2be3e8e0c8e96dd83ee, NAME => 'testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => bd66d5b12692aefad5974a6eb15a4f47, NAME => 'testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => e88d2578d5279f76f70975f8fd2b4155, NAME => 'testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => f60583096029c9557b712df1333409e7, NAME => 'testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-28 04:57:20,665 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560240663"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560240663"}]},"ts":"1677560240663"} 2023-02-28 04:57:20,665 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560240663"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560240663"}]},"ts":"1677560240663"} 2023-02-28 04:57:20,665 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560240663"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560240663"}]},"ts":"1677560240663"} 2023-02-28 04:57:20,665 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560240663"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560240663"}]},"ts":"1677560240663"} 2023-02-28 04:57:20,665 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560240663"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560240663"}]},"ts":"1677560240663"} 2023-02-28 04:57:20,665 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560240663"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560240663"}]},"ts":"1677560240663"} 2023-02-28 04:57:20,667 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-28 04:57:20,668 INFO [PEWorker-5] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-28 04:57:20,668 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 17980904c451ad53a64d5259eea8aec2, NAME => 'testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 554a5845ea85f8afefc80a4e37e38fdc, NAME => 'testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 815cdd9dbbb8a2be3e8e0c8e96dd83ee, NAME => 'testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => bd66d5b12692aefad5974a6eb15a4f47, NAME => 'testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => e88d2578d5279f76f70975f8fd2b4155, NAME => 'testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => f60583096029c9557b712df1333409e7, NAME => 'testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-28 04:57:20,668 INFO [PEWorker-5] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=emptySnaptb-1677560227585 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1677560227585 completed! 2023-02-28 04:57:20,670 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1677560227585 snapshot=name: "emptySnaptb-1677560227585" table: "testRestoreSnapshot-1677560227585" creation_time: 1677560228718 type: DISABLED version: 2 ttl: 0 ) in 49 msec 2023-02-28 04:57:20,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-28 04:57:20,874 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1677560227585, procId: 119 completed 2023-02-28 04:57:20,874 INFO [Listener at localhost/36897] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1677560227585-1677560240346 2023-02-28 04:57:20,875 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-emptySnaptb-1677560227585-1677560240346" 2023-02-28 04:57:20,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-emptySnaptb-1677560227585-1677560240346 2023-02-28 04:57:20,877 INFO [Listener at localhost/36897] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1677560227585 2023-02-28 04:57:20,877 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1677560227585 2023-02-28 04:57:20,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=120, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:20,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-28 04:57:20,881 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1677560227585 2023-02-28 04:57:20,882 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560240881"}]},"ts":"1677560240881"} 2023-02-28 04:57:20,883 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=ENABLING in hbase:meta 2023-02-28 04:57:20,885 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-28 04:57:20,885 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN}, {pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN}, {pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN}, {pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN}, {pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN}, {pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN}] 2023-02-28 04:57:20,887 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN 2023-02-28 04:57:20,887 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN 2023-02-28 04:57:20,887 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN 2023-02-28 04:57:20,887 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN 2023-02-28 04:57:20,888 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN 2023-02-28 04:57:20,888 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:20,888 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:20,888 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:20,888 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:20,889 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:20,889 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN 2023-02-28 04:57:20,890 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:21,038 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-28 04:57:21,039 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-28 04:57:21,039 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-28 04:57:21,039 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-28 04:57:21,039 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-28 04:57:21,044 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:21,044 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:21,044 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:21,044 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560241044"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560241044"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560241044"}]},"ts":"1677560241044"} 2023-02-28 04:57:21,044 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560241044"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560241044"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560241044"}]},"ts":"1677560241044"} 2023-02-28 04:57:21,044 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:21,044 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:21,044 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560241044"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560241044"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560241044"}]},"ts":"1677560241044"} 2023-02-28 04:57:21,044 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560241044"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560241044"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560241044"}]},"ts":"1677560241044"} 2023-02-28 04:57:21,044 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560241044"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560241044"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560241044"}]},"ts":"1677560241044"} 2023-02-28 04:57:21,046 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=123, state=RUNNABLE; OpenRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:21,047 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=128, ppid=124, state=RUNNABLE; OpenRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:21,047 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=125, state=RUNNABLE; OpenRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:21,048 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=126, state=RUNNABLE; OpenRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:21,049 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=122, state=RUNNABLE; OpenRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:21,050 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:21,050 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560241050"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560241050"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560241050"}]},"ts":"1677560241050"} 2023-02-28 04:57:21,054 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=132, ppid=121, state=RUNNABLE; OpenRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:21,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-28 04:57:21,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:21,200 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => bd66d5b12692aefad5974a6eb15a4f47, NAME => 'testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.', STARTKEY => '', ENDKEY => '1'} 2023-02-28 04:57:21,201 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:21,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:21,201 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:21,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:21,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f60583096029c9557b712df1333409e7, NAME => 'testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.', STARTKEY => '1', ENDKEY => '2'} 2023-02-28 04:57:21,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:21,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e88d2578d5279f76f70975f8fd2b4155, NAME => 'testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.', STARTKEY => '5', ENDKEY => ''} 2023-02-28 04:57:21,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:21,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 f60583096029c9557b712df1333409e7 2023-02-28 04:57:21,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:21,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:21,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:21,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f60583096029c9557b712df1333409e7 2023-02-28 04:57:21,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f60583096029c9557b712df1333409e7 2023-02-28 04:57:21,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:21,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:21,202 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:21,203 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f60583096029c9557b712df1333409e7 2023-02-28 04:57:21,203 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:21,204 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf 2023-02-28 04:57:21,204 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf 2023-02-28 04:57:21,204 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf 2023-02-28 04:57:21,204 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf 2023-02-28 04:57:21,204 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf 2023-02-28 04:57:21,204 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf 2023-02-28 04:57:21,204 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bd66d5b12692aefad5974a6eb15a4f47 columnFamilyName cf 2023-02-28 04:57:21,204 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f60583096029c9557b712df1333409e7 columnFamilyName cf 2023-02-28 04:57:21,204 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e88d2578d5279f76f70975f8fd2b4155 columnFamilyName cf 2023-02-28 04:57:21,205 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(310): Store=bd66d5b12692aefad5974a6eb15a4f47/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:21,206 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(310): Store=f60583096029c9557b712df1333409e7/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:21,206 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(310): Store=e88d2578d5279f76f70975f8fd2b4155/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:21,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:21,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:21,206 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:21,207 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:21,207 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:21,207 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:21,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f60583096029c9557b712df1333409e7 2023-02-28 04:57:21,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:21,210 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:21,211 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e88d2578d5279f76f70975f8fd2b4155; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74486033, jitterRate=0.10992838442325592}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:21,211 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f60583096029c9557b712df1333409e7; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73956908, jitterRate=0.10204380750656128}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:21,211 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened bd66d5b12692aefad5974a6eb15a4f47; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72983834, jitterRate=0.08754387497901917}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:21,211 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e88d2578d5279f76f70975f8fd2b4155: 2023-02-28 04:57:21,211 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for bd66d5b12692aefad5974a6eb15a4f47: 2023-02-28 04:57:21,211 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f60583096029c9557b712df1333409e7: 2023-02-28 04:57:21,212 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155., pid=130, masterSystemTime=1677560241198 2023-02-28 04:57:21,212 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47., pid=132, masterSystemTime=1677560241198 2023-02-28 04:57:21,212 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7., pid=131, masterSystemTime=1677560241199 2023-02-28 04:57:21,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:21,213 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:21,213 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:21,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 554a5845ea85f8afefc80a4e37e38fdc, NAME => 'testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.', STARTKEY => '2', ENDKEY => '3'} 2023-02-28 04:57:21,213 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:21,213 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:21,213 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560241213"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560241213"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560241213"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560241213"}]},"ts":"1677560241213"} 2023-02-28 04:57:21,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:21,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:21,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:21,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:21,214 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:21,214 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:21,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 815cdd9dbbb8a2be3e8e0c8e96dd83ee, NAME => 'testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.', STARTKEY => '3', ENDKEY => '4'} 2023-02-28 04:57:21,214 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:21,214 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560241214"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560241214"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560241214"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560241214"}]},"ts":"1677560241214"} 2023-02-28 04:57:21,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:21,214 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:21,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:21,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:21,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:21,215 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:21,215 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:21,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 17980904c451ad53a64d5259eea8aec2, NAME => 'testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.', STARTKEY => '4', ENDKEY => '5'} 2023-02-28 04:57:21,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:21,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:21,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:21,215 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:21,216 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:21,216 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:21,216 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560241216"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560241216"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560241216"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560241216"}]},"ts":"1677560241216"} 2023-02-28 04:57:21,216 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:21,217 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:21,218 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=126 2023-02-28 04:57:21,218 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf 2023-02-28 04:57:21,218 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf 2023-02-28 04:57:21,218 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=126, state=SUCCESS; OpenRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,46629,1677560224813 in 167 msec 2023-02-28 04:57:21,218 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 554a5845ea85f8afefc80a4e37e38fdc columnFamilyName cf 2023-02-28 04:57:21,219 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf 2023-02-28 04:57:21,219 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf 2023-02-28 04:57:21,219 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf 2023-02-28 04:57:21,219 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 815cdd9dbbb8a2be3e8e0c8e96dd83ee columnFamilyName cf 2023-02-28 04:57:21,219 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=121 2023-02-28 04:57:21,219 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf 2023-02-28 04:57:21,219 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=121, state=SUCCESS; OpenRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,36597,1677560224857 in 163 msec 2023-02-28 04:57:21,219 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(310): Store=554a5845ea85f8afefc80a4e37e38fdc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:21,220 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN in 333 msec 2023-02-28 04:57:21,220 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 17980904c451ad53a64d5259eea8aec2 columnFamilyName cf 2023-02-28 04:57:21,220 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=122 2023-02-28 04:57:21,220 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(310): Store=815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:21,220 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=122, state=SUCCESS; OpenRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,38247,1677560224887 in 168 msec 2023-02-28 04:57:21,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:21,221 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(310): Store=17980904c451ad53a64d5259eea8aec2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:21,221 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN in 334 msec 2023-02-28 04:57:21,221 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:21,221 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN in 335 msec 2023-02-28 04:57:21,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:21,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:21,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:21,222 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:21,224 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:21,224 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:21,224 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:21,225 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 815cdd9dbbb8a2be3e8e0c8e96dd83ee; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69619981, jitterRate=0.037418559193611145}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:21,225 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 17980904c451ad53a64d5259eea8aec2; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64395707, jitterRate=-0.040429189801216125}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:21,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 815cdd9dbbb8a2be3e8e0c8e96dd83ee: 2023-02-28 04:57:21,225 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 554a5845ea85f8afefc80a4e37e38fdc; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62266284, jitterRate=-0.07216006517410278}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:21,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 17980904c451ad53a64d5259eea8aec2: 2023-02-28 04:57:21,225 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 554a5845ea85f8afefc80a4e37e38fdc: 2023-02-28 04:57:21,226 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee., pid=128, masterSystemTime=1677560241198 2023-02-28 04:57:21,226 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2., pid=129, masterSystemTime=1677560241199 2023-02-28 04:57:21,226 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc., pid=127, masterSystemTime=1677560241198 2023-02-28 04:57:21,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:21,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:21,227 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:21,227 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560241227"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560241227"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560241227"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560241227"}]},"ts":"1677560241227"} 2023-02-28 04:57:21,227 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:21,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:21,228 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:21,228 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560241228"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560241228"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560241228"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560241228"}]},"ts":"1677560241228"} 2023-02-28 04:57:21,228 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:21,228 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:21,229 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:21,229 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560241229"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560241229"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560241229"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560241229"}]},"ts":"1677560241229"} 2023-02-28 04:57:21,231 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=128, resume processing ppid=124 2023-02-28 04:57:21,231 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=128, ppid=124, state=SUCCESS; OpenRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,36597,1677560224857 in 182 msec 2023-02-28 04:57:21,231 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=125 2023-02-28 04:57:21,231 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=125, state=SUCCESS; OpenRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,38247,1677560224887 in 182 msec 2023-02-28 04:57:21,232 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=124, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN in 346 msec 2023-02-28 04:57:21,232 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=123 2023-02-28 04:57:21,232 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=123, state=SUCCESS; OpenRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,46629,1677560224813 in 184 msec 2023-02-28 04:57:21,232 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=125, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN in 346 msec 2023-02-28 04:57:21,233 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=123, resume processing ppid=120 2023-02-28 04:57:21,233 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN in 347 msec 2023-02-28 04:57:21,234 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560241234"}]},"ts":"1677560241234"} 2023-02-28 04:57:21,235 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=ENABLED in hbase:meta 2023-02-28 04:57:21,236 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1677560227585' was successfully enabled. 2023-02-28 04:57:21,238 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=120, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1677560227585 in 359 msec 2023-02-28 04:57:21,469 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-28 04:57:21,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-28 04:57:21,633 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1677560227585, procId: 120 completed 2023-02-28 04:57:21,633 DEBUG [Listener at localhost/36897] zookeeper.ReadOnlyZKClient(139): Connect 0x58227bee to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:21,638 DEBUG [Listener at localhost/36897] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a24f0e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:21,640 DEBUG [hconnection-0x231aba10-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:21,643 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59626, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:21,645 DEBUG [hconnection-0x231aba10-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:21,647 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37272, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:21,648 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-28 04:57:21,651 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-28 04:57:21,653 DEBUG [hconnection-0x231aba10-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:21,658 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44926, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:21,659 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-28 04:57:21,662 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-28 04:57:21,664 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-28 04:57:21,669 INFO [Listener at localhost/36897] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1677560227585 2023-02-28 04:57:21,670 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1677560227585 2023-02-28 04:57:21,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=133, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:21,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-28 04:57:21,673 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560241673"}]},"ts":"1677560241673"} 2023-02-28 04:57:21,674 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=DISABLING in hbase:meta 2023-02-28 04:57:21,677 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1677560227585 to state=DISABLING 2023-02-28 04:57:21,678 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN}, {pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN}, {pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN}, {pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN}, {pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN}, {pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN}] 2023-02-28 04:57:21,680 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN 2023-02-28 04:57:21,680 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN 2023-02-28 04:57:21,680 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN 2023-02-28 04:57:21,680 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN 2023-02-28 04:57:21,680 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN 2023-02-28 04:57:21,680 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:21,681 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560241680"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560241680"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560241680"}]},"ts":"1677560241680"} 2023-02-28 04:57:21,681 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:21,681 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:21,681 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560241681"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560241681"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560241681"}]},"ts":"1677560241681"} 2023-02-28 04:57:21,681 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560241681"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560241681"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560241681"}]},"ts":"1677560241681"} 2023-02-28 04:57:21,681 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:21,681 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560241681"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560241681"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560241681"}]},"ts":"1677560241681"} 2023-02-28 04:57:21,681 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:21,682 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560241681"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560241681"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560241681"}]},"ts":"1677560241681"} 2023-02-28 04:57:21,682 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=140, ppid=138, state=RUNNABLE; CloseRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:21,683 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=141, ppid=139, state=RUNNABLE; CloseRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:21,684 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=137, state=RUNNABLE; CloseRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:21,686 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=136, state=RUNNABLE; CloseRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:21,687 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=144, ppid=135, state=RUNNABLE; CloseRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:21,687 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN 2023-02-28 04:57:21,689 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:21,689 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560241689"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560241689"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560241689"}]},"ts":"1677560241689"} 2023-02-28 04:57:21,691 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=134, state=RUNNABLE; CloseRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:21,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:21,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f60583096029c9557b712df1333409e7 2023-02-28 04:57:21,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 554a5845ea85f8afefc80a4e37e38fdc, disabling compactions & flushes 2023-02-28 04:57:21,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f60583096029c9557b712df1333409e7, disabling compactions & flushes 2023-02-28 04:57:21,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:21,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:21,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:21,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:21,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. after waiting 0 ms 2023-02-28 04:57:21,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:21,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. after waiting 0 ms 2023-02-28 04:57:21,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:21,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:21,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 815cdd9dbbb8a2be3e8e0c8e96dd83ee, disabling compactions & flushes 2023-02-28 04:57:21,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:21,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:21,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. after waiting 0 ms 2023-02-28 04:57:21,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:21,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-28 04:57:21,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-28 04:57:21,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-28 04:57:21,843 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:21,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 815cdd9dbbb8a2be3e8e0c8e96dd83ee: 2023-02-28 04:57:21,843 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:21,844 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:21,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f60583096029c9557b712df1333409e7: 2023-02-28 04:57:21,844 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 554a5845ea85f8afefc80a4e37e38fdc: 2023-02-28 04:57:21,845 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:21,845 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:21,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing bd66d5b12692aefad5974a6eb15a4f47, disabling compactions & flushes 2023-02-28 04:57:21,846 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:21,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:21,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. after waiting 0 ms 2023-02-28 04:57:21,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:21,846 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=CLOSED 2023-02-28 04:57:21,846 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560241846"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560241846"}]},"ts":"1677560241846"} 2023-02-28 04:57:21,846 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f60583096029c9557b712df1333409e7 2023-02-28 04:57:21,847 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:21,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 17980904c451ad53a64d5259eea8aec2, disabling compactions & flushes 2023-02-28 04:57:21,847 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:21,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:21,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. after waiting 0 ms 2023-02-28 04:57:21,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:21,847 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=CLOSED 2023-02-28 04:57:21,848 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560241847"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560241847"}]},"ts":"1677560241847"} 2023-02-28 04:57:21,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:21,855 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:21,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-28 04:57:21,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e88d2578d5279f76f70975f8fd2b4155, disabling compactions & flushes 2023-02-28 04:57:21,856 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-28 04:57:21,856 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:21,856 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:21,856 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. after waiting 0 ms 2023-02-28 04:57:21,856 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:21,856 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=CLOSED 2023-02-28 04:57:21,856 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560241856"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560241856"}]},"ts":"1677560241856"} 2023-02-28 04:57:21,857 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:21,858 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for bd66d5b12692aefad5974a6eb15a4f47: 2023-02-28 04:57:21,858 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:21,858 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 17980904c451ad53a64d5259eea8aec2: 2023-02-28 04:57:21,858 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=137 2023-02-28 04:57:21,858 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=144, resume processing ppid=135 2023-02-28 04:57:21,858 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=137, state=SUCCESS; CloseRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,36597,1677560224857 in 164 msec 2023-02-28 04:57:21,858 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=144, ppid=135, state=SUCCESS; CloseRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,38247,1677560224887 in 162 msec 2023-02-28 04:57:21,860 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:21,860 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=CLOSED 2023-02-28 04:57:21,860 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560241860"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560241860"}]},"ts":"1677560241860"} 2023-02-28 04:57:21,860 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:21,861 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=135, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN in 181 msec 2023-02-28 04:57:21,861 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN in 181 msec 2023-02-28 04:57:21,861 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=CLOSED 2023-02-28 04:57:21,861 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560241861"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560241861"}]},"ts":"1677560241861"} 2023-02-28 04:57:21,861 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=136 2023-02-28 04:57:21,862 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=136, state=SUCCESS; CloseRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,46629,1677560224813 in 174 msec 2023-02-28 04:57:21,863 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-28 04:57:21,863 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN in 183 msec 2023-02-28 04:57:21,864 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:21,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e88d2578d5279f76f70975f8fd2b4155: 2023-02-28 04:57:21,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:21,866 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=134 2023-02-28 04:57:21,866 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=134, state=SUCCESS; CloseRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,36597,1677560224857 in 171 msec 2023-02-28 04:57:21,866 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=140, resume processing ppid=138 2023-02-28 04:57:21,867 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=140, ppid=138, state=SUCCESS; CloseRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,38247,1677560224887 in 181 msec 2023-02-28 04:57:21,867 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=CLOSED 2023-02-28 04:57:21,867 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560241867"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560241867"}]},"ts":"1677560241867"} 2023-02-28 04:57:21,868 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN in 188 msec 2023-02-28 04:57:21,868 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN in 188 msec 2023-02-28 04:57:21,870 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=141, resume processing ppid=139 2023-02-28 04:57:21,870 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=141, ppid=139, state=SUCCESS; CloseRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,46629,1677560224813 in 185 msec 2023-02-28 04:57:21,871 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=139, resume processing ppid=133 2023-02-28 04:57:21,871 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN in 192 msec 2023-02-28 04:57:21,872 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560241872"}]},"ts":"1677560241872"} 2023-02-28 04:57:21,873 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=DISABLED in hbase:meta 2023-02-28 04:57:21,875 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1677560227585 to state=DISABLED 2023-02-28 04:57:21,876 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=133, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1677560227585 in 205 msec 2023-02-28 04:57:21,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-28 04:57:21,924 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1677560227585, procId: 133 completed 2023-02-28 04:57:21,933 INFO [Listener at localhost/36897] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb1-1677560227585-1677560241933 2023-02-28 04:57:21,934 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb1-1677560227585-1677560241933 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:21,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1677560241934 (current time:1677560241934). 2023-02-28 04:57:21,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-28 04:57:21,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] zookeeper.ReadOnlyZKClient(139): Connect 0x481410b1 to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:21,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ac14421, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:21,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:21,944 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59634, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:21,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x481410b1 to 127.0.0.1:54355 2023-02-28 04:57:21,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:21,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-28 04:57:21,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb1-1677560227585-1677560241933 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:21,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:21,948 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:21,948 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=146, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE; timeout=600000, timestamp=1677560841948 2023-02-28 04:57:21,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb1-1677560227585-1677560241933 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } 2023-02-28 04:57:21,949 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb1-1677560227585-1677560241933 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1677560227585 2023-02-28 04:57:21,949 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb1-1677560227585-1677560241933 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-28 04:57:21,949 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-28 04:57:21,958 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1677560227585-1677560241933 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } 2023-02-28 04:57:21,960 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:21,960 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:21,960 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:21,960 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:21,960 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:21,960 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:21,961 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:21,961 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:21,961 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:21,961 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:21,961 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:21,961 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:21,961 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:21,961 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:21,962 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:21,963 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:21,963 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:21,963 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:21,963 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:21,964 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:21,964 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:21,996 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb1-1677560227585-1677560241933 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } as finished. 2023-02-28 04:57:21,996 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb1-1677560227585-1677560241933 2023-02-28 04:57:21,997 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1677560227585-1677560241933 2023-02-28 04:57:22,016 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1677560227585-1677560241933 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/hbase-failsafe-snaptb1-1677560227585-1677560241933 2023-02-28 04:57:22,033 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb1-1677560227585-1677560241933 of table testRestoreSnapshot-1677560227585 completed 2023-02-28 04:57:22,033 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1677560227585-1677560241933 2023-02-28 04:57:22,033 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1677560227585-1677560241933 2023-02-28 04:57:22,033 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb1-1677560227585-1677560241933 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1677560227585 at 1677560241949Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1677560227585-1677560241933 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } at 1677560241958 (+9 ms)Consolidate snapshot: hbase-failsafe-snaptb1-1677560227585-1677560241933 at 1677560241996 (+38 ms)Loading Region manifests for hbase-failsafe-snaptb1-1677560227585-1677560241933 at 1677560241997 (+1 ms)Writing data manifest for hbase-failsafe-snaptb1-1677560227585-1677560241933 at 1677560242003 (+6 ms)Verifying snapshot: hbase-failsafe-snaptb1-1677560227585-1677560241933 at 1677560242011 (+8 ms)Snapshot hbase-failsafe-snaptb1-1677560227585-1677560241933 of table testRestoreSnapshot-1677560227585 completed at 1677560242033 (+22 ms) 2023-02-28 04:57:22,035 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=146, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE 2023-02-28 04:57:22,036 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=146, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1677560227585, type=EXCLUSIVE in 88 msec 2023-02-28 04:57:22,199 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-28 04:57:22,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1677560227585-1677560241933 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 } is done 2023-02-28 04:57:22,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb1-1677560227585-1677560241933 table=testRestoreSnapshot-1677560227585 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-28 04:57:22,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=147, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1677560227585 snapshot=name: "snaptb1-1677560227585" table: "testRestoreSnapshot-1677560227585" creation_time: 1677560236484 type: DISABLED version: 2 ttl: 0 ) 2023-02-28 04:57:22,216 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(880): Restore snapshot=snaptb1-1677560227585 as table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-28 04:57:22,227 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/.tabledesc/.tableinfo.0000000004 2023-02-28 04:57:22,228 DEBUG [PEWorker-5] util.FSTableDescriptors(519): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/.tabledesc/.tableinfo.0000000003 2023-02-28 04:57:22,228 INFO [PEWorker-5] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/.tabledesc/.tableinfo.0000000004 2023-02-28 04:57:22,229 INFO [PEWorker-5] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb1-1677560227585 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } 2023-02-28 04:57:22,231 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1677560227585" table: "testRestoreSnapshot-1677560227585" creation_time: 1677560236484 type: DISABLED version: 2 ttl: 0 2023-02-28 04:57:22,231 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,237 DEBUG [PEWorker-5] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,238 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 of snapshot=snaptb1-1677560227585 to region=45463ec309e2dd18fa7bd7d422dbf66c table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,241 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 of snapshot=snaptb1-1677560227585 to region=45463ec309e2dd18fa7bd7d422dbf66c table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,244 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 of snapshot=snaptb1-1677560227585 to region=45463ec309e2dd18fa7bd7d422dbf66c table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,247 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 of snapshot=snaptb1-1677560227585 to region=45463ec309e2dd18fa7bd7d422dbf66c table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,249 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 of snapshot=snaptb1-1677560227585 to region=45463ec309e2dd18fa7bd7d422dbf66c table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,252 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 of snapshot=snaptb1-1677560227585 to region=45463ec309e2dd18fa7bd7d422dbf66c table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,255 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b of snapshot=snaptb1-1677560227585 to region=45463ec309e2dd18fa7bd7d422dbf66c table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,258 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e of snapshot=snaptb1-1677560227585 to region=45463ec309e2dd18fa7bd7d422dbf66c table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,262 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 of snapshot=snaptb1-1677560227585 to region=45463ec309e2dd18fa7bd7d422dbf66c table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,266 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 of snapshot=snaptb1-1677560227585 to region=45463ec309e2dd18fa7bd7d422dbf66c table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,269 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 of snapshot=snaptb1-1677560227585 to region=45463ec309e2dd18fa7bd7d422dbf66c table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,273 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f of snapshot=snaptb1-1677560227585 to region=45463ec309e2dd18fa7bd7d422dbf66c table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,277 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:22,277 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:22,277 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:22,277 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:22,277 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:22,277 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(223): region to restore: f60583096029c9557b712df1333409e7 2023-02-28 04:57:22,281 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 0d9ff88f20374c829118b8980be01f63 of snapshot=snaptb1-1677560227585 to region=17980904c451ad53a64d5259eea8aec2 table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,281 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 94af33cb64e64572bdc29544552b5ca3 of snapshot=snaptb1-1677560227585 to region=815cdd9dbbb8a2be3e8e0c8e96dd83ee table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,281 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink aa2a6ee6b78142d78fa00363c7ea7a6e of snapshot=snaptb1-1677560227585 to region=554a5845ea85f8afefc80a4e37e38fdc table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,281 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 1244308735a24059b25d7ce84af068c6 of snapshot=snaptb1-1677560227585 to region=f60583096029c9557b712df1333409e7 table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,282 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 70a66588039b48cf91a99eb0ba0b59af of snapshot=snaptb1-1677560227585 to region=bd66d5b12692aefad5974a6eb15a4f47 table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,282 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 7320fea4678244c4b9b33f44fe5a0756 of snapshot=snaptb1-1677560227585 to region=e88d2578d5279f76f70975f8fd2b4155 table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,297 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 15b46826798648acb28930b8ad9e7857 of snapshot=snaptb1-1677560227585 to region=f60583096029c9557b712df1333409e7 table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,297 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 9eebf518fd074c118476c5c366e7cdf1 of snapshot=snaptb1-1677560227585 to region=e88d2578d5279f76f70975f8fd2b4155 table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,298 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink b4ee3e4950594313bbb87aa56d665751 of snapshot=snaptb1-1677560227585 to region=815cdd9dbbb8a2be3e8e0c8e96dd83ee table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,298 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink ffded11160e44993a6ff131b4e9f7276 of snapshot=snaptb1-1677560227585 to region=554a5845ea85f8afefc80a4e37e38fdc table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,304 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 7c2abaa2c369456bb51f5b5e910c2efc of snapshot=snaptb1-1677560227585 to region=bd66d5b12692aefad5974a6eb15a4f47 table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,308 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink f336080ab5da417c9ad6f84197eeb6c5 of snapshot=snaptb1-1677560227585 to region=17980904c451ad53a64d5259eea8aec2 table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,317 INFO [PEWorker-5] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1677560227585" table: "testRestoreSnapshot-1677560227585" creation_time: 1677560236484 type: DISABLED version: 2 ttl: 0 2023-02-28 04:57:22,318 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560242318"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:22,318 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560242318"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:22,318 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560242318"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:22,318 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560242318"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:22,318 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560242318"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:22,318 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560242318"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:22,320 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-28 04:57:22,321 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 17980904c451ad53a64d5259eea8aec2, NAME => 'testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 554a5845ea85f8afefc80a4e37e38fdc, NAME => 'testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 815cdd9dbbb8a2be3e8e0c8e96dd83ee, NAME => 'testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => bd66d5b12692aefad5974a6eb15a4f47, NAME => 'testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => e88d2578d5279f76f70975f8fd2b4155, NAME => 'testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => f60583096029c9557b712df1333409e7, NAME => 'testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-28 04:57:22,321 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560242319"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560242319"}]},"ts":"1677560242319"} 2023-02-28 04:57:22,321 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560242319"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560242319"}]},"ts":"1677560242319"} 2023-02-28 04:57:22,321 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560242319"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560242319"}]},"ts":"1677560242319"} 2023-02-28 04:57:22,321 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560242319"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560242319"}]},"ts":"1677560242319"} 2023-02-28 04:57:22,321 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560242319"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560242319"}]},"ts":"1677560242319"} 2023-02-28 04:57:22,321 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560242319"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560242319"}]},"ts":"1677560242319"} 2023-02-28 04:57:22,323 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-28 04:57:22,323 INFO [PEWorker-5] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-28 04:57:22,323 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 17980904c451ad53a64d5259eea8aec2, NAME => 'testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 554a5845ea85f8afefc80a4e37e38fdc, NAME => 'testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 815cdd9dbbb8a2be3e8e0c8e96dd83ee, NAME => 'testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => bd66d5b12692aefad5974a6eb15a4f47, NAME => 'testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => e88d2578d5279f76f70975f8fd2b4155, NAME => 'testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => f60583096029c9557b712df1333409e7, NAME => 'testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.', STARTKEY => '1', ENDKEY => '2'}] 2023-02-28 04:57:22,323 INFO [PEWorker-5] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb1-1677560227585 table=testRestoreSnapshot-1677560227585 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1677560227585 completed! 2023-02-28 04:57:22,326 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=147, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1677560227585 snapshot=name: "snaptb1-1677560227585" table: "testRestoreSnapshot-1677560227585" creation_time: 1677560236484 type: DISABLED version: 2 ttl: 0 ) in 111 msec 2023-02-28 04:57:22,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-28 04:57:22,469 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1677560227585, procId: 147 completed 2023-02-28 04:57:22,469 INFO [Listener at localhost/36897] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb1-1677560227585-1677560241933 2023-02-28 04:57:22,469 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb1-1677560227585-1677560241933" 2023-02-28 04:57:22,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb1-1677560227585-1677560241933 2023-02-28 04:57:22,472 INFO [Listener at localhost/36897] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,472 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=148, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-28 04:57:22,476 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1677560227585 2023-02-28 04:57:22,476 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560242476"}]},"ts":"1677560242476"} 2023-02-28 04:57:22,477 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=ENABLING in hbase:meta 2023-02-28 04:57:22,481 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-28 04:57:22,481 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN}, {pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN}, {pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN}, {pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN}, {pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN}, {pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN}] 2023-02-28 04:57:22,483 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN 2023-02-28 04:57:22,483 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN 2023-02-28 04:57:22,483 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN 2023-02-28 04:57:22,483 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN 2023-02-28 04:57:22,483 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:22,484 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:22,484 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:22,484 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN 2023-02-28 04:57:22,484 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:22,484 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN 2023-02-28 04:57:22,485 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:22,485 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-28 04:57:22,634 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-28 04:57:22,634 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-28 04:57:22,634 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-28 04:57:22,634 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-28 04:57:22,634 DEBUG [jenkins-hbase4:42505] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-28 04:57:22,637 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:22,637 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:22,638 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560242637"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560242637"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560242637"}]},"ts":"1677560242637"} 2023-02-28 04:57:22,637 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:22,637 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:22,637 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:22,638 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560242637"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560242637"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560242637"}]},"ts":"1677560242637"} 2023-02-28 04:57:22,638 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560242637"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560242637"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560242637"}]},"ts":"1677560242637"} 2023-02-28 04:57:22,638 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560242637"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560242637"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560242637"}]},"ts":"1677560242637"} 2023-02-28 04:57:22,638 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560242637"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560242637"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560242637"}]},"ts":"1677560242637"} 2023-02-28 04:57:22,639 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=152, state=RUNNABLE; OpenRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:22,640 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=156, ppid=149, state=RUNNABLE; OpenRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:22,640 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=157, ppid=153, state=RUNNABLE; OpenRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:22,641 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=158, ppid=154, state=RUNNABLE; OpenRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:22,642 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=159, ppid=151, state=RUNNABLE; OpenRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:22,642 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:22,642 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560242642"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560242642"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560242642"}]},"ts":"1677560242642"} 2023-02-28 04:57:22,644 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=160, ppid=150, state=RUNNABLE; OpenRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:22,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-28 04:57:22,793 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:22,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 815cdd9dbbb8a2be3e8e0c8e96dd83ee, NAME => 'testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.', STARTKEY => '3', ENDKEY => '4'} 2023-02-28 04:57:22,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:22,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:22,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:22,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:22,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:22,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:22,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 554a5845ea85f8afefc80a4e37e38fdc, NAME => 'testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.', STARTKEY => '2', ENDKEY => '3'} 2023-02-28 04:57:22,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f60583096029c9557b712df1333409e7, NAME => 'testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.', STARTKEY => '1', ENDKEY => '2'} 2023-02-28 04:57:22,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:22,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:22,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 f60583096029c9557b712df1333409e7 2023-02-28 04:57:22,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:22,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:22,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:22,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f60583096029c9557b712df1333409e7 2023-02-28 04:57:22,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f60583096029c9557b712df1333409e7 2023-02-28 04:57:22,796 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:22,796 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f60583096029c9557b712df1333409e7 2023-02-28 04:57:22,796 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:22,797 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf 2023-02-28 04:57:22,797 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf 2023-02-28 04:57:22,797 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf 2023-02-28 04:57:22,797 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf 2023-02-28 04:57:22,797 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 815cdd9dbbb8a2be3e8e0c8e96dd83ee columnFamilyName cf 2023-02-28 04:57:22,797 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f60583096029c9557b712df1333409e7 columnFamilyName cf 2023-02-28 04:57:22,797 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf 2023-02-28 04:57:22,798 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf 2023-02-28 04:57:22,798 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 554a5845ea85f8afefc80a4e37e38fdc columnFamilyName cf 2023-02-28 04:57:22,815 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-94af33cb64e64572bdc29544552b5ca3 2023-02-28 04:57:22,819 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-1244308735a24059b25d7ce84af068c6 2023-02-28 04:57:22,819 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-aa2a6ee6b78142d78fa00363c7ea7a6e 2023-02-28 04:57:22,827 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-b4ee3e4950594313bbb87aa56d665751 2023-02-28 04:57:22,827 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(310): Store=815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:22,827 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-15b46826798648acb28930b8ad9e7857 2023-02-28 04:57:22,827 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(310): Store=f60583096029c9557b712df1333409e7/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:22,828 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-ffded11160e44993a6ff131b4e9f7276 2023-02-28 04:57:22,828 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(310): Store=554a5845ea85f8afefc80a4e37e38fdc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:22,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:22,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:22,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:22,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:22,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:22,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:22,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f60583096029c9557b712df1333409e7 2023-02-28 04:57:22,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:22,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:22,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f60583096029c9557b712df1333409e7; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66179390, jitterRate=-0.013850241899490356}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:22,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f60583096029c9557b712df1333409e7: 2023-02-28 04:57:22,834 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 815cdd9dbbb8a2be3e8e0c8e96dd83ee; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66193281, jitterRate=-0.013643249869346619}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:22,834 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 815cdd9dbbb8a2be3e8e0c8e96dd83ee: 2023-02-28 04:57:22,835 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7., pid=160, masterSystemTime=1677560242792 2023-02-28 04:57:22,835 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 554a5845ea85f8afefc80a4e37e38fdc; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73895223, jitterRate=0.10112462937831879}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:22,835 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee., pid=155, masterSystemTime=1677560242791 2023-02-28 04:57:22,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 554a5845ea85f8afefc80a4e37e38fdc: 2023-02-28 04:57:22,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc., pid=159, masterSystemTime=1677560242792 2023-02-28 04:57:22,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:22,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:22,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:22,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 17980904c451ad53a64d5259eea8aec2, NAME => 'testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.', STARTKEY => '4', ENDKEY => '5'} 2023-02-28 04:57:22,837 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:22,837 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560242836"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560242836"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560242836"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560242836"}]},"ts":"1677560242836"} 2023-02-28 04:57:22,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:22,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:22,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:22,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:22,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:22,838 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:22,838 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:22,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => bd66d5b12692aefad5974a6eb15a4f47, NAME => 'testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.', STARTKEY => '', ENDKEY => '1'} 2023-02-28 04:57:22,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:22,839 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:22,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:22,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:22,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:22,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:22,839 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:22,839 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:22,839 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560242839"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560242839"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560242839"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560242839"}]},"ts":"1677560242839"} 2023-02-28 04:57:22,839 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:22,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e88d2578d5279f76f70975f8fd2b4155, NAME => 'testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.', STARTKEY => '5', ENDKEY => ''} 2023-02-28 04:57:22,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:22,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:22,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:22,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:22,841 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf 2023-02-28 04:57:22,841 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf 2023-02-28 04:57:22,841 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:22,841 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560242841"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560242841"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560242841"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560242841"}]},"ts":"1677560242841"} 2023-02-28 04:57:22,841 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:22,842 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:22,843 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf 2023-02-28 04:57:22,843 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf 2023-02-28 04:57:22,843 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=160, resume processing ppid=150 2023-02-28 04:57:22,843 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=160, ppid=150, state=SUCCESS; OpenRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,36597,1677560224857 in 195 msec 2023-02-28 04:57:22,843 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bd66d5b12692aefad5974a6eb15a4f47 columnFamilyName cf 2023-02-28 04:57:22,844 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf 2023-02-28 04:57:22,844 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf 2023-02-28 04:57:22,844 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=152 2023-02-28 04:57:22,844 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=152, state=SUCCESS; OpenRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,46629,1677560224813 in 202 msec 2023-02-28 04:57:22,845 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e88d2578d5279f76f70975f8fd2b4155 columnFamilyName cf 2023-02-28 04:57:22,845 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN in 362 msec 2023-02-28 04:57:22,846 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=159, resume processing ppid=151 2023-02-28 04:57:22,846 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=159, ppid=151, state=SUCCESS; OpenRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,38247,1677560224887 in 201 msec 2023-02-28 04:57:22,847 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN in 363 msec 2023-02-28 04:57:22,848 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 17980904c451ad53a64d5259eea8aec2 columnFamilyName cf 2023-02-28 04:57:22,848 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=151, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN in 365 msec 2023-02-28 04:57:22,867 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-7320fea4678244c4b9b33f44fe5a0756 2023-02-28 04:57:22,869 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-70a66588039b48cf91a99eb0ba0b59af 2023-02-28 04:57:22,876 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-0d9ff88f20374c829118b8980be01f63 2023-02-28 04:57:22,881 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-9eebf518fd074c118476c5c366e7cdf1 2023-02-28 04:57:22,881 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-7c2abaa2c369456bb51f5b5e910c2efc 2023-02-28 04:57:22,881 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(310): Store=e88d2578d5279f76f70975f8fd2b4155/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:22,881 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(310): Store=bd66d5b12692aefad5974a6eb15a4f47/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:22,882 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:22,882 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:22,883 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-f336080ab5da417c9ad6f84197eeb6c5 2023-02-28 04:57:22,884 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(310): Store=17980904c451ad53a64d5259eea8aec2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:22,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:22,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:22,884 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:22,885 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:22,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:22,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:22,887 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e88d2578d5279f76f70975f8fd2b4155; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74699104, jitterRate=0.11310338973999023}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:22,887 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e88d2578d5279f76f70975f8fd2b4155: 2023-02-28 04:57:22,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:22,888 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened bd66d5b12692aefad5974a6eb15a4f47; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69006129, jitterRate=0.028271451592445374}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:22,888 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155., pid=158, masterSystemTime=1677560242792 2023-02-28 04:57:22,888 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for bd66d5b12692aefad5974a6eb15a4f47: 2023-02-28 04:57:22,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47., pid=156, masterSystemTime=1677560242791 2023-02-28 04:57:22,889 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 17980904c451ad53a64d5259eea8aec2; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59060947, jitterRate=-0.11992330849170685}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:22,889 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 17980904c451ad53a64d5259eea8aec2: 2023-02-28 04:57:22,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2., pid=157, masterSystemTime=1677560242792 2023-02-28 04:57:22,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:22,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:22,891 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:22,891 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560242891"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560242891"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560242891"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560242891"}]},"ts":"1677560242891"} 2023-02-28 04:57:22,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:22,891 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:22,891 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:22,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:22,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:22,892 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560242891"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560242891"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560242891"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560242891"}]},"ts":"1677560242891"} 2023-02-28 04:57:22,892 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:22,892 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560242892"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560242892"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560242892"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560242892"}]},"ts":"1677560242892"} 2023-02-28 04:57:22,894 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=158, resume processing ppid=154 2023-02-28 04:57:22,894 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=158, ppid=154, state=SUCCESS; OpenRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,38247,1677560224887 in 251 msec 2023-02-28 04:57:22,895 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=156, resume processing ppid=149 2023-02-28 04:57:22,895 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=156, ppid=149, state=SUCCESS; OpenRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,46629,1677560224813 in 253 msec 2023-02-28 04:57:22,896 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=157, resume processing ppid=153 2023-02-28 04:57:22,896 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN in 413 msec 2023-02-28 04:57:22,896 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=157, ppid=153, state=SUCCESS; OpenRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,36597,1677560224857 in 254 msec 2023-02-28 04:57:22,896 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN in 414 msec 2023-02-28 04:57:22,897 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=153, resume processing ppid=148 2023-02-28 04:57:22,897 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN in 415 msec 2023-02-28 04:57:22,898 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560242898"}]},"ts":"1677560242898"} 2023-02-28 04:57:22,899 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=ENABLED in hbase:meta 2023-02-28 04:57:22,901 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1677560227585' was successfully enabled. 2023-02-28 04:57:22,902 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=148, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1677560227585 in 428 msec 2023-02-28 04:57:23,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-28 04:57:23,228 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1677560227585, procId: 148 completed 2023-02-28 04:57:23,228 DEBUG [Listener at localhost/36897] zookeeper.ReadOnlyZKClient(139): Connect 0x131d42f3 to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:23,235 DEBUG [Listener at localhost/36897] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29616110, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:23,237 DEBUG [hconnection-0x49c92e86-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:23,238 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59640, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:23,242 DEBUG [hconnection-0x49c92e86-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:23,243 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44930, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:23,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,514 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x58227bee to 127.0.0.1:54355 2023-02-28 04:57:23,514 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:23,515 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e907963 to 127.0.0.1:54355 2023-02-28 04:57:23,515 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:23,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,555 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-28 04:57:23,558 DEBUG [hconnection-0x49c92e86-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:23,561 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53620, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:23,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,813 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-28 04:57:23,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:23,996 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-28 04:57:24,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,152 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1677560227585' 2023-02-28 04:57:24,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,199 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-28 04:57:24,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,368 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-28 04:57:24,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:24,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:25,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:26,413 INFO [Listener at localhost/36897] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1677560227585 2023-02-28 04:57:26,413 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1677560227585 2023-02-28 04:57:26,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=161, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:26,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-28 04:57:26,417 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560246417"}]},"ts":"1677560246417"} 2023-02-28 04:57:26,418 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=DISABLING in hbase:meta 2023-02-28 04:57:26,420 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1677560227585 to state=DISABLING 2023-02-28 04:57:26,421 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN}, {pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN}, {pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN}, {pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN}, {pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN}, {pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN}] 2023-02-28 04:57:26,423 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN 2023-02-28 04:57:26,423 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN 2023-02-28 04:57:26,423 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN 2023-02-28 04:57:26,423 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN 2023-02-28 04:57:26,423 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN 2023-02-28 04:57:26,424 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:26,424 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:26,424 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560246424"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560246424"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560246424"}]},"ts":"1677560246424"} 2023-02-28 04:57:26,424 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560246424"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560246424"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560246424"}]},"ts":"1677560246424"} 2023-02-28 04:57:26,424 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:26,424 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:26,424 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560246424"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560246424"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560246424"}]},"ts":"1677560246424"} 2023-02-28 04:57:26,424 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560246424"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560246424"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560246424"}]},"ts":"1677560246424"} 2023-02-28 04:57:26,424 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:26,425 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560246424"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560246424"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560246424"}]},"ts":"1677560246424"} 2023-02-28 04:57:26,425 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=168, ppid=163, state=RUNNABLE; CloseRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:26,426 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=169, ppid=164, state=RUNNABLE; CloseRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:26,427 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=170, ppid=167, state=RUNNABLE; CloseRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:26,428 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=171, ppid=165, state=RUNNABLE; CloseRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:26,429 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN 2023-02-28 04:57:26,429 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=172, ppid=162, state=RUNNABLE; CloseRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:26,430 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:26,430 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560246430"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560246430"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560246430"}]},"ts":"1677560246430"} 2023-02-28 04:57:26,432 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=173, ppid=166, state=RUNNABLE; CloseRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:26,578 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:26,578 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:26,580 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 17980904c451ad53a64d5259eea8aec2, disabling compactions & flushes 2023-02-28 04:57:26,580 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e88d2578d5279f76f70975f8fd2b4155, disabling compactions & flushes 2023-02-28 04:57:26,580 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:26,580 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:26,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:26,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:26,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. after waiting 0 ms 2023-02-28 04:57:26,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. after waiting 0 ms 2023-02-28 04:57:26,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:26,581 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:26,581 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:26,582 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing bd66d5b12692aefad5974a6eb15a4f47, disabling compactions & flushes 2023-02-28 04:57:26,582 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:26,582 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:26,582 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. after waiting 0 ms 2023-02-28 04:57:26,582 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:26,592 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-28 04:57:26,592 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-28 04:57:26,592 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-28 04:57:26,592 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:26,593 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:26,593 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:26,593 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 17980904c451ad53a64d5259eea8aec2: 2023-02-28 04:57:26,593 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e88d2578d5279f76f70975f8fd2b4155: 2023-02-28 04:57:26,593 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for bd66d5b12692aefad5974a6eb15a4f47: 2023-02-28 04:57:26,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:26,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f60583096029c9557b712df1333409e7 2023-02-28 04:57:26,595 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f60583096029c9557b712df1333409e7, disabling compactions & flushes 2023-02-28 04:57:26,595 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:26,595 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:26,595 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. after waiting 0 ms 2023-02-28 04:57:26,595 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:26,595 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=CLOSED 2023-02-28 04:57:26,595 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560246595"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560246595"}]},"ts":"1677560246595"} 2023-02-28 04:57:26,598 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:26,598 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:26,599 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 815cdd9dbbb8a2be3e8e0c8e96dd83ee, disabling compactions & flushes 2023-02-28 04:57:26,599 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:26,599 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:26,599 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. after waiting 0 ms 2023-02-28 04:57:26,599 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:26,600 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=CLOSED 2023-02-28 04:57:26,600 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560246600"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560246600"}]},"ts":"1677560246600"} 2023-02-28 04:57:26,600 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:26,600 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:26,601 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 554a5845ea85f8afefc80a4e37e38fdc, disabling compactions & flushes 2023-02-28 04:57:26,603 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:26,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:26,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. after waiting 0 ms 2023-02-28 04:57:26,603 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:26,603 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=CLOSED 2023-02-28 04:57:26,604 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560246603"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560246603"}]},"ts":"1677560246603"} 2023-02-28 04:57:26,610 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=173, resume processing ppid=166 2023-02-28 04:57:26,610 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=173, ppid=166, state=SUCCESS; CloseRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,36597,1677560224857 in 168 msec 2023-02-28 04:57:26,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-28 04:57:26,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-28 04:57:26,612 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:26,612 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:26,613 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=172, resume processing ppid=162 2023-02-28 04:57:26,613 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=166, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN in 190 msec 2023-02-28 04:57:26,613 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=172, ppid=162, state=SUCCESS; CloseRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,46629,1677560224813 in 180 msec 2023-02-28 04:57:26,612 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 815cdd9dbbb8a2be3e8e0c8e96dd83ee: 2023-02-28 04:57:26,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f60583096029c9557b712df1333409e7: 2023-02-28 04:57:26,613 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=170, resume processing ppid=167 2023-02-28 04:57:26,614 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=170, ppid=167, state=SUCCESS; CloseRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,38247,1677560224887 in 183 msec 2023-02-28 04:57:26,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-28 04:57:26,614 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=162, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN in 192 msec 2023-02-28 04:57:26,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f60583096029c9557b712df1333409e7 2023-02-28 04:57:26,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:26,615 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 554a5845ea85f8afefc80a4e37e38fdc: 2023-02-28 04:57:26,615 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=167, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN in 193 msec 2023-02-28 04:57:26,615 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=CLOSED 2023-02-28 04:57:26,615 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560246615"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560246615"}]},"ts":"1677560246615"} 2023-02-28 04:57:26,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:26,616 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=CLOSED 2023-02-28 04:57:26,616 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560246616"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560246616"}]},"ts":"1677560246616"} 2023-02-28 04:57:26,616 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:26,617 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=CLOSED 2023-02-28 04:57:26,617 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560246617"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560246617"}]},"ts":"1677560246617"} 2023-02-28 04:57:26,619 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=168, resume processing ppid=163 2023-02-28 04:57:26,619 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=168, ppid=163, state=SUCCESS; CloseRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,36597,1677560224857 in 192 msec 2023-02-28 04:57:26,619 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=171, resume processing ppid=165 2023-02-28 04:57:26,619 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=171, ppid=165, state=SUCCESS; CloseRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,46629,1677560224813 in 189 msec 2023-02-28 04:57:26,620 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=163, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN in 198 msec 2023-02-28 04:57:26,620 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=169, resume processing ppid=164 2023-02-28 04:57:26,621 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=165, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN in 198 msec 2023-02-28 04:57:26,621 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=169, ppid=164, state=SUCCESS; CloseRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,38247,1677560224887 in 193 msec 2023-02-28 04:57:26,622 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=164, resume processing ppid=161 2023-02-28 04:57:26,622 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=164, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN in 200 msec 2023-02-28 04:57:26,623 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560246623"}]},"ts":"1677560246623"} 2023-02-28 04:57:26,624 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=DISABLED in hbase:meta 2023-02-28 04:57:26,626 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1677560227585 to state=DISABLED 2023-02-28 04:57:26,628 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=161, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1677560227585 in 212 msec 2023-02-28 04:57:26,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-28 04:57:26,669 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1677560227585, procId: 161 completed 2023-02-28 04:57:26,673 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1677560227585 2023-02-28 04:57:26,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:26,681 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:26,683 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=174, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:26,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-28 04:57:26,692 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:26,692 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:26,692 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:26,692 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:26,692 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:26,692 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:26,694 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/recovered.edits] 2023-02-28 04:57:26,694 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/recovered.edits] 2023-02-28 04:57:26,694 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/recovered.edits] 2023-02-28 04:57:26,694 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/recovered.edits] 2023-02-28 04:57:26,695 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/recovered.edits] 2023-02-28 04:57:26,694 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/recovered.edits] 2023-02-28 04:57:26,700 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-7320fea4678244c4b9b33f44fe5a0756 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-7320fea4678244c4b9b33f44fe5a0756 2023-02-28 04:57:26,700 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-94af33cb64e64572bdc29544552b5ca3 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-94af33cb64e64572bdc29544552b5ca3 2023-02-28 04:57:26,700 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-0d9ff88f20374c829118b8980be01f63 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-0d9ff88f20374c829118b8980be01f63 2023-02-28 04:57:26,701 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-aa2a6ee6b78142d78fa00363c7ea7a6e to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-aa2a6ee6b78142d78fa00363c7ea7a6e 2023-02-28 04:57:26,701 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-70a66588039b48cf91a99eb0ba0b59af to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-70a66588039b48cf91a99eb0ba0b59af 2023-02-28 04:57:26,701 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-1244308735a24059b25d7ce84af068c6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-1244308735a24059b25d7ce84af068c6 2023-02-28 04:57:26,702 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-ffded11160e44993a6ff131b4e9f7276 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-ffded11160e44993a6ff131b4e9f7276 2023-02-28 04:57:26,702 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-9eebf518fd074c118476c5c366e7cdf1 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-9eebf518fd074c118476c5c366e7cdf1 2023-02-28 04:57:26,703 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-f336080ab5da417c9ad6f84197eeb6c5 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-f336080ab5da417c9ad6f84197eeb6c5 2023-02-28 04:57:26,703 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-b4ee3e4950594313bbb87aa56d665751 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-b4ee3e4950594313bbb87aa56d665751 2023-02-28 04:57:26,703 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-15b46826798648acb28930b8ad9e7857 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-15b46826798648acb28930b8ad9e7857 2023-02-28 04:57:26,704 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-7c2abaa2c369456bb51f5b5e910c2efc to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-7c2abaa2c369456bb51f5b5e910c2efc 2023-02-28 04:57:26,704 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-28 04:57:26,708 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/recovered.edits/27.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/recovered.edits/27.seqid 2023-02-28 04:57:26,708 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/recovered.edits/27.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/recovered.edits/27.seqid 2023-02-28 04:57:26,708 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/recovered.edits/27.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/recovered.edits/27.seqid 2023-02-28 04:57:26,709 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/recovered.edits/27.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/recovered.edits/27.seqid 2023-02-28 04:57:26,709 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:26,709 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:26,709 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/recovered.edits/27.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/recovered.edits/27.seqid 2023-02-28 04:57:26,709 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:26,710 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:26,710 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/recovered.edits/27.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/recovered.edits/27.seqid 2023-02-28 04:57:26,710 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:26,710 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:26,710 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1677560227585 regions 2023-02-28 04:57:26,711 DEBUG [PEWorker-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c 2023-02-28 04:57:26,711 DEBUG [PEWorker-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf] 2023-02-28 04:57:26,714 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 2023-02-28 04:57:26,715 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 2023-02-28 04:57:26,716 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 2023-02-28 04:57:26,717 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 2023-02-28 04:57:26,724 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 2023-02-28 04:57:26,725 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 2023-02-28 04:57:26,726 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b 2023-02-28 04:57:26,728 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e 2023-02-28 04:57:26,729 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 2023-02-28 04:57:26,730 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 2023-02-28 04:57:26,732 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 2023-02-28 04:57:26,733 DEBUG [PEWorker-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f 2023-02-28 04:57:26,733 DEBUG [PEWorker-4] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c 2023-02-28 04:57:26,736 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=174, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:26,761 WARN [PEWorker-4] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1677560227585 from hbase:meta 2023-02-28 04:57:26,763 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1677560227585' descriptor. 2023-02-28 04:57:26,764 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=174, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:26,764 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1677560227585' from region states. 2023-02-28 04:57:26,765 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560246765"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:26,765 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560246765"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:26,765 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560246765"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:26,765 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560246765"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:26,765 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560246765"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:26,765 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560246765"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:26,767 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-28 04:57:26,767 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => bd66d5b12692aefad5974a6eb15a4f47, NAME => 'testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => f60583096029c9557b712df1333409e7, NAME => 'testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 554a5845ea85f8afefc80a4e37e38fdc, NAME => 'testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 815cdd9dbbb8a2be3e8e0c8e96dd83ee, NAME => 'testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 17980904c451ad53a64d5259eea8aec2, NAME => 'testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => e88d2578d5279f76f70975f8fd2b4155, NAME => 'testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.', STARTKEY => '5', ENDKEY => ''}] 2023-02-28 04:57:26,767 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1677560227585' as deleted. 2023-02-28 04:57:26,767 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1677560246767"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:26,769 INFO [PEWorker-4] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1677560227585 state from META 2023-02-28 04:57:26,771 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(130): Finished pid=174, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:26,772 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=174, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1677560227585 in 97 msec 2023-02-28 04:57:26,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-28 04:57:26,936 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1677560227585, procId: 174 completed 2023-02-28 04:57:26,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=175, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testRestoreSnapshot-1677560227585 snapshot=name: "snaptb1-1677560227585" table: "testRestoreSnapshot-1677560227585" creation_time: 1677560236484 type: DISABLED version: 2 ttl: 0 ) 2023-02-28 04:57:26,970 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(750): Clone snapshot=snaptb1-1677560227585 as table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:26,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-28 04:57:26,985 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/.tabledesc/.tableinfo.0000000001 2023-02-28 04:57:26,987 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1677560227585" table: "testRestoreSnapshot-1677560227585" creation_time: 1677560236484 type: DISABLED version: 2 ttl: 0 2023-02-28 04:57:26,987 DEBUG [PEWorker-1] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585 2023-02-28 04:57:26,988 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:26,990 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:26,993 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:26,995 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:26,997 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:26,999 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,001 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,003 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,005 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,006 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,009 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,011 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,013 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(239): region to add: e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:27,013 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(239): region to add: 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:27,013 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(239): region to add: f60583096029c9557b712df1333409e7 2023-02-28 04:57:27,013 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(239): region to add: 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:27,013 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(239): region to add: bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:27,013 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(239): region to add: 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:27,013 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(585): clone region=e88d2578d5279f76f70975f8fd2b4155 as e88d2578d5279f76f70975f8fd2b4155 in snapshot snaptb1-1677560227585 2023-02-28 04:57:27,013 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(585): clone region=815cdd9dbbb8a2be3e8e0c8e96dd83ee as 815cdd9dbbb8a2be3e8e0c8e96dd83ee in snapshot snaptb1-1677560227585 2023-02-28 04:57:27,013 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(585): clone region=f60583096029c9557b712df1333409e7 as f60583096029c9557b712df1333409e7 in snapshot snaptb1-1677560227585 2023-02-28 04:57:27,013 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(585): clone region=17980904c451ad53a64d5259eea8aec2 as 17980904c451ad53a64d5259eea8aec2 in snapshot snaptb1-1677560227585 2023-02-28 04:57:27,013 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(585): clone region=bd66d5b12692aefad5974a6eb15a4f47 as bd66d5b12692aefad5974a6eb15a4f47 in snapshot snaptb1-1677560227585 2023-02-28 04:57:27,013 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(585): clone region=554a5845ea85f8afefc80a4e37e38fdc as 554a5845ea85f8afefc80a4e37e38fdc in snapshot snaptb1-1677560227585 2023-02-28 04:57:27,017 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(7671): creating {ENCODED => 17980904c451ad53a64d5259eea8aec2, NAME => 'testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1677560227585', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:27,017 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(7671): creating {ENCODED => bd66d5b12692aefad5974a6eb15a4f47, NAME => 'testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1677560227585', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:27,020 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(7671): creating {ENCODED => 815cdd9dbbb8a2be3e8e0c8e96dd83ee, NAME => 'testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1677560227585', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:27,020 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(7671): creating {ENCODED => f60583096029c9557b712df1333409e7, NAME => 'testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1677560227585', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:27,020 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(7671): creating {ENCODED => 554a5845ea85f8afefc80a4e37e38fdc, NAME => 'testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1677560227585', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:27,020 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7671): creating {ENCODED => e88d2578d5279f76f70975f8fd2b4155, NAME => 'testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1677560227585', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:27,056 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:27,057 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 70a66588039b48cf91a99eb0ba0b59af from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,064 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:27,064 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:27,065 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:27,065 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink aa2a6ee6b78142d78fa00363c7ea7a6e from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,065 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 0d9ff88f20374c829118b8980be01f63 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,065 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 7320fea4678244c4b9b33f44fe5a0756 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,069 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 7c2abaa2c369456bb51f5b5e910c2efc from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,069 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 9eebf518fd074c118476c5c366e7cdf1 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,069 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink ffded11160e44993a6ff131b4e9f7276 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,070 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink f336080ab5da417c9ad6f84197eeb6c5 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,072 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:27,072 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 94af33cb64e64572bdc29544552b5ca3 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,072 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:27,072 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 1244308735a24059b25d7ce84af068c6 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,073 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1603): Closing bd66d5b12692aefad5974a6eb15a4f47, disabling compactions & flushes 2023-02-28 04:57:27,073 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:27,073 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:27,073 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. after waiting 0 ms 2023-02-28 04:57:27,073 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:27,073 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:27,073 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1557): Region close journal for bd66d5b12692aefad5974a6eb15a4f47: 2023-02-28 04:57:27,074 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1603): Closing e88d2578d5279f76f70975f8fd2b4155, disabling compactions & flushes 2023-02-28 04:57:27,074 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:27,074 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:27,074 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. after waiting 0 ms 2023-02-28 04:57:27,074 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:27,074 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:27,074 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1557): Region close journal for e88d2578d5279f76f70975f8fd2b4155: 2023-02-28 04:57:27,076 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1603): Closing 17980904c451ad53a64d5259eea8aec2, disabling compactions & flushes 2023-02-28 04:57:27,076 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:27,076 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:27,076 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. after waiting 0 ms 2023-02-28 04:57:27,076 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:27,076 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:27,076 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1557): Region close journal for 17980904c451ad53a64d5259eea8aec2: 2023-02-28 04:57:27,076 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1603): Closing 554a5845ea85f8afefc80a4e37e38fdc, disabling compactions & flushes 2023-02-28 04:57:27,077 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:27,077 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:27,077 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. after waiting 0 ms 2023-02-28 04:57:27,077 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:27,077 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:27,077 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1557): Region close journal for 554a5845ea85f8afefc80a4e37e38fdc: 2023-02-28 04:57:27,077 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink b4ee3e4950594313bbb87aa56d665751 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,077 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 15b46826798648acb28930b8ad9e7857 from cloned region in snapshot snaptb1-1677560227585 to table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:27,079 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1603): Closing 815cdd9dbbb8a2be3e8e0c8e96dd83ee, disabling compactions & flushes 2023-02-28 04:57:27,079 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:27,079 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:27,079 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1603): Closing f60583096029c9557b712df1333409e7, disabling compactions & flushes 2023-02-28 04:57:27,079 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. after waiting 0 ms 2023-02-28 04:57:27,079 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:27,079 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:27,079 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:27,079 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:27,079 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1557): Region close journal for 815cdd9dbbb8a2be3e8e0c8e96dd83ee: 2023-02-28 04:57:27,079 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. after waiting 0 ms 2023-02-28 04:57:27,079 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:27,079 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:27,079 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1557): Region close journal for f60583096029c9557b712df1333409e7: 2023-02-28 04:57:27,080 INFO [PEWorker-1] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1677560227585" table: "testRestoreSnapshot-1677560227585" creation_time: 1677560236484 type: DISABLED version: 2 ttl: 0 2023-02-28 04:57:27,080 INFO [PEWorker-1] procedure.CloneSnapshotProcedure$1(411): Clone snapshot=snaptb1-1677560227585 on table=testRestoreSnapshot-1677560227585 completed! 2023-02-28 04:57:27,084 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560247084"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560247084"}]},"ts":"1677560247084"} 2023-02-28 04:57:27,084 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560247084"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560247084"}]},"ts":"1677560247084"} 2023-02-28 04:57:27,084 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560247084"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560247084"}]},"ts":"1677560247084"} 2023-02-28 04:57:27,084 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560247084"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560247084"}]},"ts":"1677560247084"} 2023-02-28 04:57:27,084 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560247084"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560247084"}]},"ts":"1677560247084"} 2023-02-28 04:57:27,084 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560247084"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560247084"}]},"ts":"1677560247084"} 2023-02-28 04:57:27,086 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-28 04:57:27,087 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560247087"}]},"ts":"1677560247087"} 2023-02-28 04:57:27,088 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=ENABLING in hbase:meta 2023-02-28 04:57:27,094 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-28 04:57:27,094 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-28 04:57:27,094 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-28 04:57:27,094 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-28 04:57:27,094 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-28 04:57:27,095 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN}, {pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN}, {pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN}, {pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN}, {pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN}, {pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN}] 2023-02-28 04:57:27,097 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN 2023-02-28 04:57:27,097 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN 2023-02-28 04:57:27,097 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN 2023-02-28 04:57:27,097 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN 2023-02-28 04:57:27,097 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN 2023-02-28 04:57:27,097 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38247,1677560224887; forceNewPlan=false, retain=false 2023-02-28 04:57:27,098 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46629,1677560224813; forceNewPlan=false, retain=false 2023-02-28 04:57:27,098 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46629,1677560224813; forceNewPlan=false, retain=false 2023-02-28 04:57:27,098 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36597,1677560224857; forceNewPlan=false, retain=false 2023-02-28 04:57:27,098 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38247,1677560224887; forceNewPlan=false, retain=false 2023-02-28 04:57:27,098 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN 2023-02-28 04:57:27,099 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36597,1677560224857; forceNewPlan=false, retain=false 2023-02-28 04:57:27,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-28 04:57:27,248 INFO [jenkins-hbase4:42505] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-28 04:57:27,252 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:27,252 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:27,252 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560247252"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560247252"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560247252"}]},"ts":"1677560247252"} 2023-02-28 04:57:27,252 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:27,252 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:27,252 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:27,252 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560247252"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560247252"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560247252"}]},"ts":"1677560247252"} 2023-02-28 04:57:27,252 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560247252"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560247252"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560247252"}]},"ts":"1677560247252"} 2023-02-28 04:57:27,252 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560247252"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560247252"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560247252"}]},"ts":"1677560247252"} 2023-02-28 04:57:27,252 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560247252"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560247252"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560247252"}]},"ts":"1677560247252"} 2023-02-28 04:57:27,254 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=182, ppid=180, state=RUNNABLE; OpenRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:27,256 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=183, ppid=176, state=RUNNABLE; OpenRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:27,257 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=184, ppid=181, state=RUNNABLE; OpenRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:27,258 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=185, ppid=178, state=RUNNABLE; OpenRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:27,258 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:27,258 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560247258"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560247258"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560247258"}]},"ts":"1677560247258"} 2023-02-28 04:57:27,258 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=186, ppid=177, state=RUNNABLE; OpenRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:27,260 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=187, ppid=179, state=RUNNABLE; OpenRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:27,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:27,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 815cdd9dbbb8a2be3e8e0c8e96dd83ee, NAME => 'testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.', STARTKEY => '3', ENDKEY => '4'} 2023-02-28 04:57:27,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:27,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:27,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:27,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 17980904c451ad53a64d5259eea8aec2, NAME => 'testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.', STARTKEY => '4', ENDKEY => '5'} 2023-02-28 04:57:27,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:27,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:27,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:27,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:27,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:27,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:27,411 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:27,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 554a5845ea85f8afefc80a4e37e38fdc, NAME => 'testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.', STARTKEY => '2', ENDKEY => '3'} 2023-02-28 04:57:27,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:27,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:27,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:27,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:27,412 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:27,413 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:27,413 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:27,414 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf 2023-02-28 04:57:27,414 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf 2023-02-28 04:57:27,414 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf 2023-02-28 04:57:27,414 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf 2023-02-28 04:57:27,414 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf 2023-02-28 04:57:27,414 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf 2023-02-28 04:57:27,414 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 554a5845ea85f8afefc80a4e37e38fdc columnFamilyName cf 2023-02-28 04:57:27,414 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 815cdd9dbbb8a2be3e8e0c8e96dd83ee columnFamilyName cf 2023-02-28 04:57:27,414 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 17980904c451ad53a64d5259eea8aec2 columnFamilyName cf 2023-02-28 04:57:27,426 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-94af33cb64e64572bdc29544552b5ca3 2023-02-28 04:57:27,428 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-aa2a6ee6b78142d78fa00363c7ea7a6e 2023-02-28 04:57:27,428 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-0d9ff88f20374c829118b8980be01f63 2023-02-28 04:57:27,434 DEBUG [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-b4ee3e4950594313bbb87aa56d665751 2023-02-28 04:57:27,434 INFO [StoreOpener-815cdd9dbbb8a2be3e8e0c8e96dd83ee-1] regionserver.HStore(310): Store=815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:27,434 DEBUG [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-f336080ab5da417c9ad6f84197eeb6c5 2023-02-28 04:57:27,434 INFO [StoreOpener-17980904c451ad53a64d5259eea8aec2-1] regionserver.HStore(310): Store=17980904c451ad53a64d5259eea8aec2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:27,435 DEBUG [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-ffded11160e44993a6ff131b4e9f7276 2023-02-28 04:57:27,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:27,435 INFO [StoreOpener-554a5845ea85f8afefc80a4e37e38fdc-1] regionserver.HStore(310): Store=554a5845ea85f8afefc80a4e37e38fdc/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:27,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:27,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:27,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:27,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:27,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:27,438 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:27,438 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:27,438 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:27,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-28 04:57:27,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-28 04:57:27,440 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 554a5845ea85f8afefc80a4e37e38fdc; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70516571, jitterRate=0.050778791308403015}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:27,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-28 04:57:27,441 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 554a5845ea85f8afefc80a4e37e38fdc: 2023-02-28 04:57:27,441 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 815cdd9dbbb8a2be3e8e0c8e96dd83ee; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63594763, jitterRate=-0.05236418545246124}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:27,441 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 815cdd9dbbb8a2be3e8e0c8e96dd83ee: 2023-02-28 04:57:27,441 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 17980904c451ad53a64d5259eea8aec2; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66816362, jitterRate=-0.004358619451522827}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:27,441 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 17980904c451ad53a64d5259eea8aec2: 2023-02-28 04:57:27,441 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc., pid=185, masterSystemTime=1677560247409 2023-02-28 04:57:27,441 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee., pid=187, masterSystemTime=1677560247407 2023-02-28 04:57:27,442 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2., pid=182, masterSystemTime=1677560247407 2023-02-28 04:57:27,443 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:27,443 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:27,443 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:27,443 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f60583096029c9557b712df1333409e7, NAME => 'testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.', STARTKEY => '1', ENDKEY => '2'} 2023-02-28 04:57:27,443 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:27,443 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 f60583096029c9557b712df1333409e7 2023-02-28 04:57:27,443 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560247443"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560247443"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560247443"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560247443"}]},"ts":"1677560247443"} 2023-02-28 04:57:27,443 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:27,443 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f60583096029c9557b712df1333409e7 2023-02-28 04:57:27,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f60583096029c9557b712df1333409e7 2023-02-28 04:57:27,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:27,444 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:27,444 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:27,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => bd66d5b12692aefad5974a6eb15a4f47, NAME => 'testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.', STARTKEY => '', ENDKEY => '1'} 2023-02-28 04:57:27,444 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:27,444 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560247444"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560247444"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560247444"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560247444"}]},"ts":"1677560247444"} 2023-02-28 04:57:27,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:27,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:27,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:27,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:27,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:27,445 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:27,445 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:27,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => e88d2578d5279f76f70975f8fd2b4155, NAME => 'testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.', STARTKEY => '5', ENDKEY => ''} 2023-02-28 04:57:27,445 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f60583096029c9557b712df1333409e7 2023-02-28 04:57:27,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1677560227585 e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:27,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:27,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:27,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:27,445 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:27,446 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:27,446 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560247445"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560247445"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560247445"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560247445"}]},"ts":"1677560247445"} 2023-02-28 04:57:27,446 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf 2023-02-28 04:57:27,446 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf 2023-02-28 04:57:27,446 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf 2023-02-28 04:57:27,446 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf 2023-02-28 04:57:27,447 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f60583096029c9557b712df1333409e7 columnFamilyName cf 2023-02-28 04:57:27,447 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bd66d5b12692aefad5974a6eb15a4f47 columnFamilyName cf 2023-02-28 04:57:27,447 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:27,448 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=185, resume processing ppid=178 2023-02-28 04:57:27,448 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=185, ppid=178, state=SUCCESS; OpenRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,38247,1677560224887 in 189 msec 2023-02-28 04:57:27,449 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf 2023-02-28 04:57:27,449 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf 2023-02-28 04:57:27,449 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e88d2578d5279f76f70975f8fd2b4155 columnFamilyName cf 2023-02-28 04:57:27,450 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=178, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, ASSIGN in 353 msec 2023-02-28 04:57:27,451 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=187, resume processing ppid=179 2023-02-28 04:57:27,451 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=187, ppid=179, state=SUCCESS; OpenRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,36597,1677560224857 in 188 msec 2023-02-28 04:57:27,453 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=182, resume processing ppid=180 2023-02-28 04:57:27,453 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=182, ppid=180, state=SUCCESS; OpenRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,46629,1677560224813 in 195 msec 2023-02-28 04:57:27,455 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=179, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, ASSIGN in 356 msec 2023-02-28 04:57:27,464 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=180, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, ASSIGN in 358 msec 2023-02-28 04:57:27,464 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-70a66588039b48cf91a99eb0ba0b59af 2023-02-28 04:57:27,464 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-7320fea4678244c4b9b33f44fe5a0756 2023-02-28 04:57:27,466 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-1244308735a24059b25d7ce84af068c6 2023-02-28 04:57:27,468 DEBUG [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-9eebf518fd074c118476c5c366e7cdf1 2023-02-28 04:57:27,468 INFO [StoreOpener-e88d2578d5279f76f70975f8fd2b4155-1] regionserver.HStore(310): Store=e88d2578d5279f76f70975f8fd2b4155/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:27,468 DEBUG [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-7c2abaa2c369456bb51f5b5e910c2efc 2023-02-28 04:57:27,469 INFO [StoreOpener-bd66d5b12692aefad5974a6eb15a4f47-1] regionserver.HStore(310): Store=bd66d5b12692aefad5974a6eb15a4f47/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:27,469 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:27,469 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:27,469 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:27,470 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:27,470 DEBUG [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-15b46826798648acb28930b8ad9e7857 2023-02-28 04:57:27,470 INFO [StoreOpener-f60583096029c9557b712df1333409e7-1] regionserver.HStore(310): Store=f60583096029c9557b712df1333409e7/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:27,471 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:27,471 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:27,472 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:27,472 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:27,473 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f60583096029c9557b712df1333409e7 2023-02-28 04:57:27,473 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-28 04:57:27,474 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-28 04:57:27,474 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened e88d2578d5279f76f70975f8fd2b4155; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62566563, jitterRate=-0.0676855593919754}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:27,474 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for e88d2578d5279f76f70975f8fd2b4155: 2023-02-28 04:57:27,474 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened bd66d5b12692aefad5974a6eb15a4f47; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60592366, jitterRate=-0.09710338711738586}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:27,474 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for bd66d5b12692aefad5974a6eb15a4f47: 2023-02-28 04:57:27,475 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155., pid=184, masterSystemTime=1677560247407 2023-02-28 04:57:27,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-28 04:57:27,475 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47., pid=183, masterSystemTime=1677560247407 2023-02-28 04:57:27,475 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f60583096029c9557b712df1333409e7; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72665245, jitterRate=0.08279652893543243}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:27,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f60583096029c9557b712df1333409e7: 2023-02-28 04:57:27,476 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:27,476 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7., pid=186, masterSystemTime=1677560247409 2023-02-28 04:57:27,476 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:27,477 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:27,477 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:27,477 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560247476"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560247476"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560247476"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560247476"}]},"ts":"1677560247476"} 2023-02-28 04:57:27,477 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:27,477 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:27,477 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560247477"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560247477"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560247477"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560247477"}]},"ts":"1677560247477"} 2023-02-28 04:57:27,477 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:27,478 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:27,478 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:27,478 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560247478"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560247478"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560247478"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560247478"}]},"ts":"1677560247478"} 2023-02-28 04:57:27,480 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=184, resume processing ppid=181 2023-02-28 04:57:27,480 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=184, ppid=181, state=SUCCESS; OpenRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,46629,1677560224813 in 221 msec 2023-02-28 04:57:27,480 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=183, resume processing ppid=176 2023-02-28 04:57:27,481 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=183, ppid=176, state=SUCCESS; OpenRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,36597,1677560224857 in 225 msec 2023-02-28 04:57:27,481 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=181, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, ASSIGN in 385 msec 2023-02-28 04:57:27,482 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=186, resume processing ppid=177 2023-02-28 04:57:27,482 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=186, ppid=177, state=SUCCESS; OpenRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,38247,1677560224887 in 222 msec 2023-02-28 04:57:27,482 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=176, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, ASSIGN in 386 msec 2023-02-28 04:57:27,483 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=177, resume processing ppid=175 2023-02-28 04:57:27,483 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=177, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, ASSIGN in 387 msec 2023-02-28 04:57:27,484 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560247484"}]},"ts":"1677560247484"} 2023-02-28 04:57:27,485 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=ENABLED in hbase:meta 2023-02-28 04:57:27,489 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=175, state=SUCCESS; CloneSnapshotProcedure (table=testRestoreSnapshot-1677560227585 snapshot=name: "snaptb1-1677560227585" table: "testRestoreSnapshot-1677560227585" creation_time: 1677560236484 type: DISABLED version: 2 ttl: 0 ) in 523 msec 2023-02-28 04:57:27,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-28 04:57:27,724 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1677560227585, procId: 175 completed 2023-02-28 04:57:27,725 DEBUG [Listener at localhost/36897] zookeeper.ReadOnlyZKClient(139): Connect 0x594888d9 to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:27,733 DEBUG [Listener at localhost/36897] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7254d96e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:27,736 DEBUG [hconnection-0x1913e027-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:27,737 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41620, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:27,740 DEBUG [hconnection-0x1913e027-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:27,741 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53632, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:27,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,977 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-28 04:57:27,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:27,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,175 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-28 04:57:28,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,322 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-28 04:57:28,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,493 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-28 04:57:28,496 DEBUG [hconnection-0x1913e027-shared-pool-4] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:28,497 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40042, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:28,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,666 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-28 04:57:28,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:28,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,226 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x131d42f3 to 127.0.0.1:54355 2023-02-28 04:57:29,226 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:29,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:29,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:30,444 INFO [Listener at localhost/36897] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1677560227585 2023-02-28 04:57:30,444 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1677560227585 2023-02-28 04:57:30,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=188, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:30,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-28 04:57:30,448 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560250448"}]},"ts":"1677560250448"} 2023-02-28 04:57:30,449 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=DISABLING in hbase:meta 2023-02-28 04:57:30,451 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1677560227585 to state=DISABLING 2023-02-28 04:57:30,451 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN}, {pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN}, {pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN}, {pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN}, {pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN}, {pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN}] 2023-02-28 04:57:30,453 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN 2023-02-28 04:57:30,453 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN 2023-02-28 04:57:30,453 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN 2023-02-28 04:57:30,453 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN 2023-02-28 04:57:30,453 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN 2023-02-28 04:57:30,454 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:30,454 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560250454"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560250454"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560250454"}]},"ts":"1677560250454"} 2023-02-28 04:57:30,454 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:30,454 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:30,454 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560250454"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560250454"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560250454"}]},"ts":"1677560250454"} 2023-02-28 04:57:30,454 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560250454"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560250454"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560250454"}]},"ts":"1677560250454"} 2023-02-28 04:57:30,454 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:30,454 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560250454"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560250454"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560250454"}]},"ts":"1677560250454"} 2023-02-28 04:57:30,455 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:30,455 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560250454"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560250454"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560250454"}]},"ts":"1677560250454"} 2023-02-28 04:57:30,455 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=195, ppid=191, state=RUNNABLE; CloseRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:30,456 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=196, ppid=189, state=RUNNABLE; CloseRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:30,457 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=197, ppid=190, state=RUNNABLE; CloseRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:30,460 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=198, ppid=194, state=RUNNABLE; CloseRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:30,461 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN 2023-02-28 04:57:30,461 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=199, ppid=192, state=RUNNABLE; CloseRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:30,462 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:30,462 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560250462"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560250462"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560250462"}]},"ts":"1677560250462"} 2023-02-28 04:57:30,463 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=200, ppid=193, state=RUNNABLE; CloseRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:30,608 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:30,609 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 554a5845ea85f8afefc80a4e37e38fdc, disabling compactions & flushes 2023-02-28 04:57:30,609 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:30,609 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:30,609 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. after waiting 0 ms 2023-02-28 04:57:30,609 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:30,609 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:30,610 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 815cdd9dbbb8a2be3e8e0c8e96dd83ee, disabling compactions & flushes 2023-02-28 04:57:30,610 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:30,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:30,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. after waiting 0 ms 2023-02-28 04:57:30,613 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:30,614 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:30,614 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 17980904c451ad53a64d5259eea8aec2, disabling compactions & flushes 2023-02-28 04:57:30,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:30,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:30,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. after waiting 0 ms 2023-02-28 04:57:30,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:30,622 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-28 04:57:30,622 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-28 04:57:30,622 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc. 2023-02-28 04:57:30,622 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee. 2023-02-28 04:57:30,622 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 554a5845ea85f8afefc80a4e37e38fdc: 2023-02-28 04:57:30,622 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 815cdd9dbbb8a2be3e8e0c8e96dd83ee: 2023-02-28 04:57:30,623 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-28 04:57:30,624 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2. 2023-02-28 04:57:30,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 17980904c451ad53a64d5259eea8aec2: 2023-02-28 04:57:30,624 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:30,624 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:30,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing bd66d5b12692aefad5974a6eb15a4f47, disabling compactions & flushes 2023-02-28 04:57:30,624 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:30,624 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:30,625 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. after waiting 0 ms 2023-02-28 04:57:30,625 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:30,625 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=815cdd9dbbb8a2be3e8e0c8e96dd83ee, regionState=CLOSED 2023-02-28 04:57:30,626 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560250625"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560250625"}]},"ts":"1677560250625"} 2023-02-28 04:57:30,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:30,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f60583096029c9557b712df1333409e7 2023-02-28 04:57:30,632 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f60583096029c9557b712df1333409e7, disabling compactions & flushes 2023-02-28 04:57:30,632 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:30,632 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:30,632 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. after waiting 0 ms 2023-02-28 04:57:30,632 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:30,633 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=554a5845ea85f8afefc80a4e37e38fdc, regionState=CLOSED 2023-02-28 04:57:30,633 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560250633"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560250633"}]},"ts":"1677560250633"} 2023-02-28 04:57:30,634 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:30,634 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:30,635 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing e88d2578d5279f76f70975f8fd2b4155, disabling compactions & flushes 2023-02-28 04:57:30,635 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:30,635 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:30,635 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. after waiting 0 ms 2023-02-28 04:57:30,635 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:30,635 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=17980904c451ad53a64d5259eea8aec2, regionState=CLOSED 2023-02-28 04:57:30,635 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560250635"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560250635"}]},"ts":"1677560250635"} 2023-02-28 04:57:30,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-28 04:57:30,638 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-28 04:57:30,638 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47. 2023-02-28 04:57:30,638 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for bd66d5b12692aefad5974a6eb15a4f47: 2023-02-28 04:57:30,639 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=199, resume processing ppid=192 2023-02-28 04:57:30,639 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7. 2023-02-28 04:57:30,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f60583096029c9557b712df1333409e7: 2023-02-28 04:57:30,639 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=199, ppid=192, state=SUCCESS; CloseRegionProcedure 815cdd9dbbb8a2be3e8e0c8e96dd83ee, server=jenkins-hbase4.apache.org,36597,1677560224857 in 175 msec 2023-02-28 04:57:30,639 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=195, resume processing ppid=191 2023-02-28 04:57:30,639 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=195, ppid=191, state=SUCCESS; CloseRegionProcedure 554a5845ea85f8afefc80a4e37e38fdc, server=jenkins-hbase4.apache.org,38247,1677560224887 in 182 msec 2023-02-28 04:57:30,640 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=200, resume processing ppid=193 2023-02-28 04:57:30,640 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=200, ppid=193, state=SUCCESS; CloseRegionProcedure 17980904c451ad53a64d5259eea8aec2, server=jenkins-hbase4.apache.org,46629,1677560224813 in 175 msec 2023-02-28 04:57:30,640 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:30,640 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-28 04:57:30,640 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=192, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=815cdd9dbbb8a2be3e8e0c8e96dd83ee, UNASSIGN in 188 msec 2023-02-28 04:57:30,641 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=bd66d5b12692aefad5974a6eb15a4f47, regionState=CLOSED 2023-02-28 04:57:30,641 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=191, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=554a5845ea85f8afefc80a4e37e38fdc, UNASSIGN in 188 msec 2023-02-28 04:57:30,641 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560250641"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560250641"}]},"ts":"1677560250641"} 2023-02-28 04:57:30,641 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f60583096029c9557b712df1333409e7 2023-02-28 04:57:30,641 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155. 2023-02-28 04:57:30,641 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for e88d2578d5279f76f70975f8fd2b4155: 2023-02-28 04:57:30,641 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=193, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=17980904c451ad53a64d5259eea8aec2, UNASSIGN in 189 msec 2023-02-28 04:57:30,641 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=f60583096029c9557b712df1333409e7, regionState=CLOSED 2023-02-28 04:57:30,642 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1677560250641"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560250641"}]},"ts":"1677560250641"} 2023-02-28 04:57:30,642 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:30,643 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=e88d2578d5279f76f70975f8fd2b4155, regionState=CLOSED 2023-02-28 04:57:30,643 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1677560250643"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560250643"}]},"ts":"1677560250643"} 2023-02-28 04:57:30,644 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=196, resume processing ppid=189 2023-02-28 04:57:30,644 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=196, ppid=189, state=SUCCESS; CloseRegionProcedure bd66d5b12692aefad5974a6eb15a4f47, server=jenkins-hbase4.apache.org,36597,1677560224857 in 186 msec 2023-02-28 04:57:30,645 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=197, resume processing ppid=190 2023-02-28 04:57:30,645 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=197, ppid=190, state=SUCCESS; CloseRegionProcedure f60583096029c9557b712df1333409e7, server=jenkins-hbase4.apache.org,38247,1677560224887 in 186 msec 2023-02-28 04:57:30,646 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=189, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=bd66d5b12692aefad5974a6eb15a4f47, UNASSIGN in 193 msec 2023-02-28 04:57:30,646 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=190, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=f60583096029c9557b712df1333409e7, UNASSIGN in 194 msec 2023-02-28 04:57:30,647 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=198, resume processing ppid=194 2023-02-28 04:57:30,647 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=198, ppid=194, state=SUCCESS; CloseRegionProcedure e88d2578d5279f76f70975f8fd2b4155, server=jenkins-hbase4.apache.org,46629,1677560224813 in 187 msec 2023-02-28 04:57:30,649 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=194, resume processing ppid=188 2023-02-28 04:57:30,649 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=194, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1677560227585, region=e88d2578d5279f76f70975f8fd2b4155, UNASSIGN in 196 msec 2023-02-28 04:57:30,649 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560250649"}]},"ts":"1677560250649"} 2023-02-28 04:57:30,650 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1677560227585, state=DISABLED in hbase:meta 2023-02-28 04:57:30,652 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1677560227585 to state=DISABLED 2023-02-28 04:57:30,654 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=188, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1677560227585 in 208 msec 2023-02-28 04:57:30,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-28 04:57:30,699 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1677560227585, procId: 188 completed 2023-02-28 04:57:30,700 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1677560227585 2023-02-28 04:57:30,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:30,702 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:30,702 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=201, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:30,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-28 04:57:30,709 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:30,709 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:30,709 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:30,709 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:30,709 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:30,709 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:30,711 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/recovered.edits] 2023-02-28 04:57:30,711 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/recovered.edits] 2023-02-28 04:57:30,711 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/recovered.edits] 2023-02-28 04:57:30,711 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/recovered.edits] 2023-02-28 04:57:30,711 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/recovered.edits] 2023-02-28 04:57:30,712 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/recovered.edits] 2023-02-28 04:57:30,714 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-aa2a6ee6b78142d78fa00363c7ea7a6e already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-aa2a6ee6b78142d78fa00363c7ea7a6e. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246699 currentMtime: 1677560247069 2023-02-28 04:57:30,715 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-70a66588039b48cf91a99eb0ba0b59af already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-70a66588039b48cf91a99eb0ba0b59af. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246700 currentMtime: 1677560247068 2023-02-28 04:57:30,715 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-aa2a6ee6b78142d78fa00363c7ea7a6e to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-aa2a6ee6b78142d78fa00363c7ea7a6e.1677560250711. 2023-02-28 04:57:30,715 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-1244308735a24059b25d7ce84af068c6 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-1244308735a24059b25d7ce84af068c6. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246699 currentMtime: 1677560247077 2023-02-28 04:57:30,715 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-70a66588039b48cf91a99eb0ba0b59af to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-70a66588039b48cf91a99eb0ba0b59af.1677560250711. 2023-02-28 04:57:30,716 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-94af33cb64e64572bdc29544552b5ca3 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-94af33cb64e64572bdc29544552b5ca3. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246698 currentMtime: 1677560247077 2023-02-28 04:57:30,716 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-0d9ff88f20374c829118b8980be01f63 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-0d9ff88f20374c829118b8980be01f63. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246698 currentMtime: 1677560247070 2023-02-28 04:57:30,716 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-aa2a6ee6b78142d78fa00363c7ea7a6e to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-aa2a6ee6b78142d78fa00363c7ea7a6e 2023-02-28 04:57:30,716 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-7320fea4678244c4b9b33f44fe5a0756 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-7320fea4678244c4b9b33f44fe5a0756. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246698 currentMtime: 1677560247069 2023-02-28 04:57:30,716 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-1244308735a24059b25d7ce84af068c6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-1244308735a24059b25d7ce84af068c6.1677560250711. 2023-02-28 04:57:30,716 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-94af33cb64e64572bdc29544552b5ca3 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-94af33cb64e64572bdc29544552b5ca3.1677560250712. 2023-02-28 04:57:30,716 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-0d9ff88f20374c829118b8980be01f63 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-0d9ff88f20374c829118b8980be01f63.1677560250711. 2023-02-28 04:57:30,717 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-70a66588039b48cf91a99eb0ba0b59af to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-70a66588039b48cf91a99eb0ba0b59af 2023-02-28 04:57:30,717 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-7320fea4678244c4b9b33f44fe5a0756 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-7320fea4678244c4b9b33f44fe5a0756.1677560250711. 2023-02-28 04:57:30,717 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-94af33cb64e64572bdc29544552b5ca3 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-94af33cb64e64572bdc29544552b5ca3 2023-02-28 04:57:30,717 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-0d9ff88f20374c829118b8980be01f63 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-0d9ff88f20374c829118b8980be01f63 2023-02-28 04:57:30,717 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-1244308735a24059b25d7ce84af068c6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-1244308735a24059b25d7ce84af068c6 2023-02-28 04:57:30,718 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-7320fea4678244c4b9b33f44fe5a0756 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-7320fea4678244c4b9b33f44fe5a0756 2023-02-28 04:57:30,718 ERROR [HFileArchiver-5] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-ffded11160e44993a6ff131b4e9f7276 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-ffded11160e44993a6ff131b4e9f7276. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246701 currentMtime: 1677560247076 2023-02-28 04:57:30,719 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-7c2abaa2c369456bb51f5b5e910c2efc already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-7c2abaa2c369456bb51f5b5e910c2efc. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246702 currentMtime: 1677560247072 2023-02-28 04:57:30,719 INFO [HFileArchiver-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-ffded11160e44993a6ff131b4e9f7276 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-ffded11160e44993a6ff131b4e9f7276.1677560250711. 2023-02-28 04:57:30,719 ERROR [HFileArchiver-7] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-15b46826798648acb28930b8ad9e7857 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-15b46826798648acb28930b8ad9e7857. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246701 currentMtime: 1677560247079 2023-02-28 04:57:30,719 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-b4ee3e4950594313bbb87aa56d665751 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-b4ee3e4950594313bbb87aa56d665751. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246701 currentMtime: 1677560247079 2023-02-28 04:57:30,719 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-7c2abaa2c369456bb51f5b5e910c2efc to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-7c2abaa2c369456bb51f5b5e910c2efc.1677560250711. 2023-02-28 04:57:30,720 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-b4ee3e4950594313bbb87aa56d665751 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-b4ee3e4950594313bbb87aa56d665751.1677560250712. 2023-02-28 04:57:30,720 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-9eebf518fd074c118476c5c366e7cdf1 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-9eebf518fd074c118476c5c366e7cdf1. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246701 currentMtime: 1677560247073 2023-02-28 04:57:30,720 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-f336080ab5da417c9ad6f84197eeb6c5 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-f336080ab5da417c9ad6f84197eeb6c5. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246701 currentMtime: 1677560247075 2023-02-28 04:57:30,720 INFO [HFileArchiver-7] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-15b46826798648acb28930b8ad9e7857 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-15b46826798648acb28930b8ad9e7857.1677560250711. 2023-02-28 04:57:30,720 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-ffded11160e44993a6ff131b4e9f7276 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/cf/testRestoreSnapshot-1677560227585=554a5845ea85f8afefc80a4e37e38fdc-ffded11160e44993a6ff131b4e9f7276 2023-02-28 04:57:30,720 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-9eebf518fd074c118476c5c366e7cdf1 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-9eebf518fd074c118476c5c366e7cdf1.1677560250711. 2023-02-28 04:57:30,720 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-f336080ab5da417c9ad6f84197eeb6c5 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-f336080ab5da417c9ad6f84197eeb6c5.1677560250711. 2023-02-28 04:57:30,721 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-7c2abaa2c369456bb51f5b5e910c2efc to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/cf/testRestoreSnapshot-1677560227585=bd66d5b12692aefad5974a6eb15a4f47-7c2abaa2c369456bb51f5b5e910c2efc 2023-02-28 04:57:30,721 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-b4ee3e4950594313bbb87aa56d665751 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/cf/testRestoreSnapshot-1677560227585=815cdd9dbbb8a2be3e8e0c8e96dd83ee-b4ee3e4950594313bbb87aa56d665751 2023-02-28 04:57:30,721 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-15b46826798648acb28930b8ad9e7857 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/cf/testRestoreSnapshot-1677560227585=f60583096029c9557b712df1333409e7-15b46826798648acb28930b8ad9e7857 2023-02-28 04:57:30,721 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-f336080ab5da417c9ad6f84197eeb6c5 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/cf/testRestoreSnapshot-1677560227585=17980904c451ad53a64d5259eea8aec2-f336080ab5da417c9ad6f84197eeb6c5 2023-02-28 04:57:30,721 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-9eebf518fd074c118476c5c366e7cdf1 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/cf/testRestoreSnapshot-1677560227585=e88d2578d5279f76f70975f8fd2b4155-9eebf518fd074c118476c5c366e7cdf1 2023-02-28 04:57:30,724 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/recovered.edits/18.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc/recovered.edits/18.seqid 2023-02-28 04:57:30,724 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/recovered.edits/18.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47/recovered.edits/18.seqid 2023-02-28 04:57:30,724 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/recovered.edits/18.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee/recovered.edits/18.seqid 2023-02-28 04:57:30,724 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/recovered.edits/18.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2/recovered.edits/18.seqid 2023-02-28 04:57:30,725 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/recovered.edits/18.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7/recovered.edits/18.seqid 2023-02-28 04:57:30,725 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/recovered.edits/18.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155/recovered.edits/18.seqid 2023-02-28 04:57:30,725 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/554a5845ea85f8afefc80a4e37e38fdc 2023-02-28 04:57:30,725 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/17980904c451ad53a64d5259eea8aec2 2023-02-28 04:57:30,726 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/815cdd9dbbb8a2be3e8e0c8e96dd83ee 2023-02-28 04:57:30,726 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/bd66d5b12692aefad5974a6eb15a4f47 2023-02-28 04:57:30,726 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/f60583096029c9557b712df1333409e7 2023-02-28 04:57:30,726 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testRestoreSnapshot-1677560227585/e88d2578d5279f76f70975f8fd2b4155 2023-02-28 04:57:30,726 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1677560227585 regions 2023-02-28 04:57:30,727 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c 2023-02-28 04:57:30,727 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf] 2023-02-28 04:57:30,729 ERROR [PEWorker-5] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246713 currentMtime: 1677560246990 2023-02-28 04:57:30,729 INFO [PEWorker-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436.1677560250727. 2023-02-28 04:57:30,730 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-a87ff679a2f3e71d9181a67b7542122c202302289062c99ebdd841b4b7f14b56a2772436 2023-02-28 04:57:30,730 ERROR [PEWorker-5] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246714 currentMtime: 1677560246992 2023-02-28 04:57:30,731 INFO [PEWorker-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6.1677560250727. 2023-02-28 04:57:30,731 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-a87ff679a2f3e71d9181a67b7542122c20230228de7159473d3c4b2caf275f3c5cc3d7c6 2023-02-28 04:57:30,732 ERROR [PEWorker-5] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246715 currentMtime: 1677560246994 2023-02-28 04:57:30,733 INFO [PEWorker-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787.1677560250727. 2023-02-28 04:57:30,733 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c4ca4238a0b923820dcc509a6f75849b2023022883df2079d5b24535bbd2be87840b4787 2023-02-28 04:57:30,734 ERROR [PEWorker-5] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246716 currentMtime: 1677560246996 2023-02-28 04:57:30,734 INFO [PEWorker-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6.1677560250727. 2023-02-28 04:57:30,735 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c4ca4238a0b923820dcc509a6f75849b20230228efbd175a3dc04b0d94d22cb8b44f87b6 2023-02-28 04:57:30,736 ERROR [PEWorker-5] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246723 currentMtime: 1677560246999 2023-02-28 04:57:30,736 INFO [PEWorker-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93.1677560250727. 2023-02-28 04:57:30,737 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c81e728d9d4c2f636f067f89cc14862c20230228014f5775d8ea4ad49f661b052ea4ea93 2023-02-28 04:57:30,738 ERROR [PEWorker-5] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246724 currentMtime: 1677560247001 2023-02-28 04:57:30,738 INFO [PEWorker-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9.1677560250727. 2023-02-28 04:57:30,739 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-c81e728d9d4c2f636f067f89cc14862c202302284f2f4bfa3815483296e2e319a2ac0dc9 2023-02-28 04:57:30,740 ERROR [PEWorker-5] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246726 currentMtime: 1677560247003 2023-02-28 04:57:30,740 INFO [PEWorker-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b.1677560250727. 2023-02-28 04:57:30,741 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-d41d8cd98f00b204e9800998ecf8427e20230228e7a97ec99eaa488ca783d667dc21780b 2023-02-28 04:57:30,742 ERROR [PEWorker-5] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246727 currentMtime: 1677560247004 2023-02-28 04:57:30,742 INFO [PEWorker-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e.1677560250727. 2023-02-28 04:57:30,743 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-d41d8cd98f00b204e9800998ecf8427e20230228f164e79da0384869ae5fb6299167353e 2023-02-28 04:57:30,744 ERROR [PEWorker-5] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246728 currentMtime: 1677560247006 2023-02-28 04:57:30,744 INFO [PEWorker-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80.1677560250727. 2023-02-28 04:57:30,744 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-e4da3b7fbbce2345d7772b0674a318d520230228028d170be7384cdbb1bf1e241d5cfb80 2023-02-28 04:57:30,746 ERROR [PEWorker-5] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246730 currentMtime: 1677560247009 2023-02-28 04:57:30,746 INFO [PEWorker-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884.1677560250727. 2023-02-28 04:57:30,746 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-e4da3b7fbbce2345d7772b0674a318d520230228cdfbfce040f241b392703df7a3904884 2023-02-28 04:57:30,747 ERROR [PEWorker-5] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246731 currentMtime: 1677560247010 2023-02-28 04:57:30,748 INFO [PEWorker-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963.1677560250727. 2023-02-28 04:57:30,748 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-eccbc87e4b5ce2fe28308fd9f2a7baf3202302283a281cd42b0c4f2fac5b080673520963 2023-02-28 04:57:30,749 ERROR [PEWorker-5] backup.HFileArchiver(523): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f. archiveLen: 0 currentLen: 0 archiveMtime: 1677560246732 currentMtime: 1677560247013 2023-02-28 04:57:30,750 INFO [PEWorker-5] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f.1677560250727. 2023-02-28 04:57:30,750 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c/cf/testRestoreSnapshot-1677560227585=45463ec309e2dd18fa7bd7d422dbf66c-eccbc87e4b5ce2fe28308fd9f2a7baf3202302289da398d6fb61430e86cbcaa40db85e7f 2023-02-28 04:57:30,751 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testRestoreSnapshot-1677560227585/45463ec309e2dd18fa7bd7d422dbf66c 2023-02-28 04:57:30,753 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=201, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:30,755 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1677560227585 from hbase:meta 2023-02-28 04:57:30,757 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1677560227585' descriptor. 2023-02-28 04:57:30,758 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=201, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:30,758 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1677560227585' from region states. 2023-02-28 04:57:30,758 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560250758"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:30,758 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560250758"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:30,758 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560250758"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:30,758 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560250758"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:30,758 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560250758"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:30,758 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560250758"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:30,760 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-28 04:57:30,760 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => bd66d5b12692aefad5974a6eb15a4f47, NAME => 'testRestoreSnapshot-1677560227585,,1677560227607.bd66d5b12692aefad5974a6eb15a4f47.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => f60583096029c9557b712df1333409e7, NAME => 'testRestoreSnapshot-1677560227585,1,1677560227607.f60583096029c9557b712df1333409e7.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 554a5845ea85f8afefc80a4e37e38fdc, NAME => 'testRestoreSnapshot-1677560227585,2,1677560227607.554a5845ea85f8afefc80a4e37e38fdc.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 815cdd9dbbb8a2be3e8e0c8e96dd83ee, NAME => 'testRestoreSnapshot-1677560227585,3,1677560227607.815cdd9dbbb8a2be3e8e0c8e96dd83ee.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 17980904c451ad53a64d5259eea8aec2, NAME => 'testRestoreSnapshot-1677560227585,4,1677560227607.17980904c451ad53a64d5259eea8aec2.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => e88d2578d5279f76f70975f8fd2b4155, NAME => 'testRestoreSnapshot-1677560227585,5,1677560227607.e88d2578d5279f76f70975f8fd2b4155.', STARTKEY => '5', ENDKEY => ''}] 2023-02-28 04:57:30,760 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1677560227585' as deleted. 2023-02-28 04:57:30,760 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1677560227585","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1677560250760"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:30,761 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1677560227585 state from META 2023-02-28 04:57:30,763 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=201, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1677560227585 2023-02-28 04:57:30,764 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=201, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1677560227585 in 63 msec 2023-02-28 04:57:30,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-28 04:57:30,954 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1677560227585, procId: 201 completed 2023-02-28 04:57:30,960 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1677560227585" 2023-02-28 04:57:30,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1677560227585 2023-02-28 04:57:30,962 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1677560227585" 2023-02-28 04:57:30,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1677560227585 2023-02-28 04:57:30,964 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb1-1677560227585" 2023-02-28 04:57:30,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(333): Deleting snapshot: snaptb1-1677560227585 2023-02-28 04:57:30,993 INFO [Listener at localhost/36897] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=431 (was 361) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5d8659e3-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1029099772_15 at /127.0.0.1:49818 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/cluster_b26b0118-f74c-2753-b135-766a0fc054b5/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:54355@0x594888d9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$45/1706886088.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/cluster_b26b0118-f74c-2753-b135-766a0fc054b5/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-metaLookup-shared--pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-920385773_15 at /127.0.0.1:39544 [Waiting for operation #486] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1029099772_15 at /127.0.0.1:39510 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1841729027_15 at /127.0.0.1:49732 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-920385773_15 at /127.0.0.1:35228 [Waiting for operation #521] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:54355@0x594888d9-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: hconnection-0x5d8659e3-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-shared-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/cluster_b26b0118-f74c-2753-b135-766a0fc054b5/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-4-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5d8659e3-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x56d8d559-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_2118870683_15 at /127.0.0.1:39532 [Waiting for operation #493] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-920385773_15 at /127.0.0.1:40878 [Waiting for operation #502] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/cluster_b26b0118-f74c-2753-b135-766a0fc054b5/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-shared-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_2118870683_15 at /127.0.0.1:39436 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-metaLookup-shared--pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-920385773_15 at /127.0.0.1:40870 [Waiting for operation #504] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_2118870683_15 at /127.0.0.1:49828 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1029099772_15 at /127.0.0.1:37562 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/cluster_b26b0118-f74c-2753-b135-766a0fc054b5/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: hconnection-0x1913e027-shared-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/cluster_b26b0118-f74c-2753-b135-766a0fc054b5/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-metaLookup-shared--pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-shared-pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x5d8659e3-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1841729027_15 at /127.0.0.1:39520 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-metaLookup-shared--pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-920385773_15 at /127.0.0.1:37568 [Waiting for operation #501] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x1913e027-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1841729027_15 at /127.0.0.1:39500 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_2118870683_15 at /127.0.0.1:39514 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-920385773_15 at /127.0.0.1:52768 [Waiting for operation #1530] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:54355@0x594888d9-SendThread(127.0.0.1:54355) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) - Thread LEAK? -, OpenFileDescriptor=818 (was 688) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=204 (was 227), ProcessCount=170 (was 173), AvailableMemoryMB=8873 (was 9332) 2023-02-28 04:57:31,008 INFO [Listener at localhost/36897] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=429, OpenFileDescriptor=818, MaxFileDescriptor=60000, SystemLoadAverage=204, ProcessCount=170, AvailableMemoryMB=8872 2023-02-28 04:57:31,011 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'testCorruptedSnapshot-1677560251009', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-28 04:57:31,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCorruptedSnapshot-1677560251009 2023-02-28 04:57:31,013 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1677560251009 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-28 04:57:31,014 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testCorruptedSnapshot-1677560251009" procId is: 202 2023-02-28 04:57:31,014 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1677560251009 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-28 04:57:31,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-28 04:57:31,019 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:31,019 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:31,019 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e 2023-02-28 04:57:31,019 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:31,019 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:31,019 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:31,020 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81 empty. 2023-02-28 04:57:31,020 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6 empty. 2023-02-28 04:57:31,020 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db empty. 2023-02-28 04:57:31,020 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda empty. 2023-02-28 04:57:31,020 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e empty. 2023-02-28 04:57:31,020 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d empty. 2023-02-28 04:57:31,020 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:31,020 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:31,020 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:31,021 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:31,021 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e 2023-02-28 04:57:31,021 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:31,021 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1677560251009 regions 2023-02-28 04:57:31,034 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/.tabledesc/.tableinfo.0000000001 2023-02-28 04:57:31,037 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(7671): creating {ENCODED => 830ed62397c5222f09d35b7149e1b7db, NAME => 'testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testCorruptedSnapshot-1677560251009', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:31,037 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(7671): creating {ENCODED => d67d3cb3ce05e60ff75f17313b64811d, NAME => 'testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testCorruptedSnapshot-1677560251009', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:31,037 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(7671): creating {ENCODED => 9bc02e4052125a0d5af36b95e5b25a81, NAME => 'testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testCorruptedSnapshot-1677560251009', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:31,052 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:31,052 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(1603): Closing 830ed62397c5222f09d35b7149e1b7db, disabling compactions & flushes 2023-02-28 04:57:31,052 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:31,052 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:31,052 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. after waiting 0 ms 2023-02-28 04:57:31,052 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:31,052 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:31,052 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(1557): Region close journal for 830ed62397c5222f09d35b7149e1b7db: 2023-02-28 04:57:31,053 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(7671): creating {ENCODED => 640ee1a6dbf01ebb011a298d972247c6, NAME => 'testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testCorruptedSnapshot-1677560251009', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:31,054 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:31,055 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(1603): Closing 9bc02e4052125a0d5af36b95e5b25a81, disabling compactions & flushes 2023-02-28 04:57:31,055 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:31,055 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:31,055 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. after waiting 0 ms 2023-02-28 04:57:31,055 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:31,055 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:31,055 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(1557): Region close journal for 9bc02e4052125a0d5af36b95e5b25a81: 2023-02-28 04:57:31,056 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(7671): creating {ENCODED => 6d657e535457f1d48270505f48f3ceda, NAME => 'testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testCorruptedSnapshot-1677560251009', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:31,056 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:31,056 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(1603): Closing d67d3cb3ce05e60ff75f17313b64811d, disabling compactions & flushes 2023-02-28 04:57:31,056 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:31,057 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:31,057 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. after waiting 0 ms 2023-02-28 04:57:31,057 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:31,057 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:31,057 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(1557): Region close journal for d67d3cb3ce05e60ff75f17313b64811d: 2023-02-28 04:57:31,057 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(7671): creating {ENCODED => 35c83130614b8300868b867ad906a41e, NAME => 'testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testCorruptedSnapshot-1677560251009', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp 2023-02-28 04:57:31,079 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:31,079 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:31,079 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(1603): Closing 640ee1a6dbf01ebb011a298d972247c6, disabling compactions & flushes 2023-02-28 04:57:31,079 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(1603): Closing 6d657e535457f1d48270505f48f3ceda, disabling compactions & flushes 2023-02-28 04:57:31,079 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:31,079 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:31,079 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:31,079 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:31,079 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. after waiting 0 ms 2023-02-28 04:57:31,079 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. after waiting 0 ms 2023-02-28 04:57:31,079 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:31,079 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:31,079 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:31,079 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:31,079 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-1] regionserver.HRegion(1557): Region close journal for 640ee1a6dbf01ebb011a298d972247c6: 2023-02-28 04:57:31,079 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-0] regionserver.HRegion(1557): Region close journal for 6d657e535457f1d48270505f48f3ceda: 2023-02-28 04:57:31,080 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:31,080 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(1603): Closing 35c83130614b8300868b867ad906a41e, disabling compactions & flushes 2023-02-28 04:57:31,080 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:31,080 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:31,080 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. after waiting 0 ms 2023-02-28 04:57:31,080 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:31,080 INFO [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:31,080 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1677560251009-pool-2] regionserver.HRegion(1557): Region close journal for 35c83130614b8300868b867ad906a41e: 2023-02-28 04:57:31,082 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1677560251009 execute state=CREATE_TABLE_ADD_TO_META 2023-02-28 04:57:31,083 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251083"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560251083"}]},"ts":"1677560251083"} 2023-02-28 04:57:31,083 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560251083"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560251083"}]},"ts":"1677560251083"} 2023-02-28 04:57:31,083 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251083"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560251083"}]},"ts":"1677560251083"} 2023-02-28 04:57:31,083 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251083"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560251083"}]},"ts":"1677560251083"} 2023-02-28 04:57:31,083 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251083"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560251083"}]},"ts":"1677560251083"} 2023-02-28 04:57:31,083 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560251083"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560251083"}]},"ts":"1677560251083"} 2023-02-28 04:57:31,088 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-28 04:57:31,088 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1677560251009 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-28 04:57:31,089 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560251088"}]},"ts":"1677560251088"} 2023-02-28 04:57:31,090 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1677560251009, state=ENABLING in hbase:meta 2023-02-28 04:57:31,095 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-28 04:57:31,097 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-28 04:57:31,097 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-28 04:57:31,097 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-28 04:57:31,097 DEBUG [PEWorker-4] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-28 04:57:31,097 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, ASSIGN}, {pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, ASSIGN}, {pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, ASSIGN}, {pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, ASSIGN}, {pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, ASSIGN}, {pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, ASSIGN}] 2023-02-28 04:57:31,099 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, ASSIGN 2023-02-28 04:57:31,099 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, ASSIGN 2023-02-28 04:57:31,099 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, ASSIGN 2023-02-28 04:57:31,099 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, ASSIGN 2023-02-28 04:57:31,099 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38247,1677560224887; forceNewPlan=false, retain=false 2023-02-28 04:57:31,099 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46629,1677560224813; forceNewPlan=false, retain=false 2023-02-28 04:57:31,099 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36597,1677560224857; forceNewPlan=false, retain=false 2023-02-28 04:57:31,099 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36597,1677560224857; forceNewPlan=false, retain=false 2023-02-28 04:57:31,099 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, ASSIGN 2023-02-28 04:57:31,100 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, ASSIGN 2023-02-28 04:57:31,100 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,46629,1677560224813; forceNewPlan=false, retain=false 2023-02-28 04:57:31,101 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38247,1677560224887; forceNewPlan=false, retain=false 2023-02-28 04:57:31,249 INFO [jenkins-hbase4:42505] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-28 04:57:31,254 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=35c83130614b8300868b867ad906a41e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:31,254 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=830ed62397c5222f09d35b7149e1b7db, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:31,254 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560251254"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560251254"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560251254"}]},"ts":"1677560251254"} 2023-02-28 04:57:31,254 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=6d657e535457f1d48270505f48f3ceda, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:31,254 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=9bc02e4052125a0d5af36b95e5b25a81, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:31,254 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251254"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560251254"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560251254"}]},"ts":"1677560251254"} 2023-02-28 04:57:31,254 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560251254"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560251254"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560251254"}]},"ts":"1677560251254"} 2023-02-28 04:57:31,254 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=d67d3cb3ce05e60ff75f17313b64811d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:31,254 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251254"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560251254"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560251254"}]},"ts":"1677560251254"} 2023-02-28 04:57:31,255 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251254"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560251254"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560251254"}]},"ts":"1677560251254"} 2023-02-28 04:57:31,255 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=209, ppid=208, state=RUNNABLE; OpenRegionProcedure 35c83130614b8300868b867ad906a41e, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:31,256 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=210, ppid=207, state=RUNNABLE; OpenRegionProcedure 6d657e535457f1d48270505f48f3ceda, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:31,257 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=211, ppid=203, state=RUNNABLE; OpenRegionProcedure 9bc02e4052125a0d5af36b95e5b25a81, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:31,257 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=212, ppid=204, state=RUNNABLE; OpenRegionProcedure 830ed62397c5222f09d35b7149e1b7db, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:31,258 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=213, ppid=205, state=RUNNABLE; OpenRegionProcedure d67d3cb3ce05e60ff75f17313b64811d, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:31,259 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=640ee1a6dbf01ebb011a298d972247c6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:31,259 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251259"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560251259"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560251259"}]},"ts":"1677560251259"} 2023-02-28 04:57:31,260 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=214, ppid=206, state=RUNNABLE; OpenRegionProcedure 640ee1a6dbf01ebb011a298d972247c6, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:31,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-28 04:57:31,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:31,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d67d3cb3ce05e60ff75f17313b64811d, NAME => 'testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.', STARTKEY => '2', ENDKEY => '3'} 2023-02-28 04:57:31,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:31,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:31,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:31,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:31,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:31,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6d657e535457f1d48270505f48f3ceda, NAME => 'testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.', STARTKEY => '4', ENDKEY => '5'} 2023-02-28 04:57:31,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:31,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:31,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:31,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:31,411 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:31,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 830ed62397c5222f09d35b7149e1b7db, NAME => 'testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.', STARTKEY => '1', ENDKEY => '2'} 2023-02-28 04:57:31,411 INFO [StoreOpener-d67d3cb3ce05e60ff75f17313b64811d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:31,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:31,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:31,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:31,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:31,412 INFO [StoreOpener-6d657e535457f1d48270505f48f3ceda-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:31,413 INFO [StoreOpener-830ed62397c5222f09d35b7149e1b7db-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:31,413 DEBUG [StoreOpener-d67d3cb3ce05e60ff75f17313b64811d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf 2023-02-28 04:57:31,413 DEBUG [StoreOpener-d67d3cb3ce05e60ff75f17313b64811d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf 2023-02-28 04:57:31,413 INFO [StoreOpener-d67d3cb3ce05e60ff75f17313b64811d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d67d3cb3ce05e60ff75f17313b64811d columnFamilyName cf 2023-02-28 04:57:31,413 DEBUG [StoreOpener-6d657e535457f1d48270505f48f3ceda-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf 2023-02-28 04:57:31,414 DEBUG [StoreOpener-6d657e535457f1d48270505f48f3ceda-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf 2023-02-28 04:57:31,414 DEBUG [StoreOpener-830ed62397c5222f09d35b7149e1b7db-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf 2023-02-28 04:57:31,414 DEBUG [StoreOpener-830ed62397c5222f09d35b7149e1b7db-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf 2023-02-28 04:57:31,414 INFO [StoreOpener-6d657e535457f1d48270505f48f3ceda-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6d657e535457f1d48270505f48f3ceda columnFamilyName cf 2023-02-28 04:57:31,414 INFO [StoreOpener-830ed62397c5222f09d35b7149e1b7db-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 830ed62397c5222f09d35b7149e1b7db columnFamilyName cf 2023-02-28 04:57:31,414 INFO [StoreOpener-d67d3cb3ce05e60ff75f17313b64811d-1] regionserver.HStore(310): Store=d67d3cb3ce05e60ff75f17313b64811d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:31,415 INFO [StoreOpener-6d657e535457f1d48270505f48f3ceda-1] regionserver.HStore(310): Store=6d657e535457f1d48270505f48f3ceda/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:31,415 INFO [StoreOpener-830ed62397c5222f09d35b7149e1b7db-1] regionserver.HStore(310): Store=830ed62397c5222f09d35b7149e1b7db/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:31,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:31,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:31,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:31,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:31,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:31,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:31,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:31,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:31,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:31,419 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-28 04:57:31,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-28 04:57:31,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-28 04:57:31,420 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 6d657e535457f1d48270505f48f3ceda; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58874841, jitterRate=-0.12269650399684906}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:31,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 6d657e535457f1d48270505f48f3ceda: 2023-02-28 04:57:31,420 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 830ed62397c5222f09d35b7149e1b7db; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75186850, jitterRate=0.12037137150764465}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:31,420 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d67d3cb3ce05e60ff75f17313b64811d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72932071, jitterRate=0.08677254617214203}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:31,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 830ed62397c5222f09d35b7149e1b7db: 2023-02-28 04:57:31,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d67d3cb3ce05e60ff75f17313b64811d: 2023-02-28 04:57:31,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d., pid=213, masterSystemTime=1677560251407 2023-02-28 04:57:31,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db., pid=212, masterSystemTime=1677560251410 2023-02-28 04:57:31,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda., pid=210, masterSystemTime=1677560251408 2023-02-28 04:57:31,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:31,422 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:31,422 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:31,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 35c83130614b8300868b867ad906a41e, NAME => 'testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.', STARTKEY => '5', ENDKEY => ''} 2023-02-28 04:57:31,422 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=d67d3cb3ce05e60ff75f17313b64811d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:31,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:31,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:31,423 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251422"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560251422"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560251422"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560251422"}]},"ts":"1677560251422"} 2023-02-28 04:57:31,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:31,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:31,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:31,423 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:31,423 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:31,423 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=6d657e535457f1d48270505f48f3ceda, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:31,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9bc02e4052125a0d5af36b95e5b25a81, NAME => 'testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.', STARTKEY => '', ENDKEY => '1'} 2023-02-28 04:57:31,423 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251423"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560251423"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560251423"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560251423"}]},"ts":"1677560251423"} 2023-02-28 04:57:31,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:31,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:31,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:31,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:31,424 INFO [StoreOpener-35c83130614b8300868b867ad906a41e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:31,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:31,424 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:31,424 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:31,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 640ee1a6dbf01ebb011a298d972247c6, NAME => 'testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.', STARTKEY => '3', ENDKEY => '4'} 2023-02-28 04:57:31,424 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=830ed62397c5222f09d35b7149e1b7db, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:31,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:31,424 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251424"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560251424"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560251424"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560251424"}]},"ts":"1677560251424"} 2023-02-28 04:57:31,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:31,425 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:31,425 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:31,424 INFO [StoreOpener-9bc02e4052125a0d5af36b95e5b25a81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:31,426 DEBUG [StoreOpener-35c83130614b8300868b867ad906a41e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf 2023-02-28 04:57:31,426 DEBUG [StoreOpener-35c83130614b8300868b867ad906a41e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf 2023-02-28 04:57:31,426 DEBUG [StoreOpener-9bc02e4052125a0d5af36b95e5b25a81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf 2023-02-28 04:57:31,426 DEBUG [StoreOpener-9bc02e4052125a0d5af36b95e5b25a81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf 2023-02-28 04:57:31,426 INFO [StoreOpener-640ee1a6dbf01ebb011a298d972247c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:31,426 INFO [StoreOpener-35c83130614b8300868b867ad906a41e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 35c83130614b8300868b867ad906a41e columnFamilyName cf 2023-02-28 04:57:31,426 INFO [StoreOpener-9bc02e4052125a0d5af36b95e5b25a81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9bc02e4052125a0d5af36b95e5b25a81 columnFamilyName cf 2023-02-28 04:57:31,427 INFO [StoreOpener-35c83130614b8300868b867ad906a41e-1] regionserver.HStore(310): Store=35c83130614b8300868b867ad906a41e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:31,427 INFO [StoreOpener-9bc02e4052125a0d5af36b95e5b25a81-1] regionserver.HStore(310): Store=9bc02e4052125a0d5af36b95e5b25a81/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:31,427 DEBUG [StoreOpener-640ee1a6dbf01ebb011a298d972247c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf 2023-02-28 04:57:31,427 DEBUG [StoreOpener-640ee1a6dbf01ebb011a298d972247c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf 2023-02-28 04:57:31,427 INFO [StoreOpener-640ee1a6dbf01ebb011a298d972247c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 640ee1a6dbf01ebb011a298d972247c6 columnFamilyName cf 2023-02-28 04:57:31,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e 2023-02-28 04:57:31,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:31,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e 2023-02-28 04:57:31,428 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=213, resume processing ppid=205 2023-02-28 04:57:31,428 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=213, ppid=205, state=SUCCESS; OpenRegionProcedure d67d3cb3ce05e60ff75f17313b64811d, server=jenkins-hbase4.apache.org,46629,1677560224813 in 166 msec 2023-02-28 04:57:31,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:31,429 INFO [StoreOpener-640ee1a6dbf01ebb011a298d972247c6-1] regionserver.HStore(310): Store=640ee1a6dbf01ebb011a298d972247c6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:31,429 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=210, resume processing ppid=207 2023-02-28 04:57:31,429 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=210, ppid=207, state=SUCCESS; OpenRegionProcedure 6d657e535457f1d48270505f48f3ceda, server=jenkins-hbase4.apache.org,38247,1677560224887 in 169 msec 2023-02-28 04:57:31,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:31,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:31,431 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=205, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, ASSIGN in 331 msec 2023-02-28 04:57:31,431 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=212, resume processing ppid=204 2023-02-28 04:57:31,431 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=212, ppid=204, state=SUCCESS; OpenRegionProcedure 830ed62397c5222f09d35b7149e1b7db, server=jenkins-hbase4.apache.org,36597,1677560224857 in 171 msec 2023-02-28 04:57:31,431 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=207, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, ASSIGN in 332 msec 2023-02-28 04:57:31,431 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:31,431 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:31,432 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=204, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, ASSIGN in 334 msec 2023-02-28 04:57:31,433 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:31,433 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-28 04:57:31,433 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-28 04:57:31,434 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9bc02e4052125a0d5af36b95e5b25a81; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66847664, jitterRate=-0.003892183303833008}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:31,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9bc02e4052125a0d5af36b95e5b25a81: 2023-02-28 04:57:31,434 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 35c83130614b8300868b867ad906a41e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65696449, jitterRate=-0.021046623587608337}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:31,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 35c83130614b8300868b867ad906a41e: 2023-02-28 04:57:31,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-28 04:57:31,434 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81., pid=211, masterSystemTime=1677560251408 2023-02-28 04:57:31,434 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e., pid=209, masterSystemTime=1677560251407 2023-02-28 04:57:31,435 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 640ee1a6dbf01ebb011a298d972247c6; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61557132, jitterRate=-0.08272725343704224}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:31,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 640ee1a6dbf01ebb011a298d972247c6: 2023-02-28 04:57:31,435 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6., pid=214, masterSystemTime=1677560251410 2023-02-28 04:57:31,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:31,436 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:31,436 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=9bc02e4052125a0d5af36b95e5b25a81, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:31,436 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560251436"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560251436"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560251436"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560251436"}]},"ts":"1677560251436"} 2023-02-28 04:57:31,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:31,436 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:31,437 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=35c83130614b8300868b867ad906a41e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:31,437 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560251437"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560251437"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560251437"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560251437"}]},"ts":"1677560251437"} 2023-02-28 04:57:31,437 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:31,437 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:31,437 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=640ee1a6dbf01ebb011a298d972247c6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:31,437 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251437"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560251437"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560251437"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560251437"}]},"ts":"1677560251437"} 2023-02-28 04:57:31,439 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=211, resume processing ppid=203 2023-02-28 04:57:31,439 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=211, ppid=203, state=SUCCESS; OpenRegionProcedure 9bc02e4052125a0d5af36b95e5b25a81, server=jenkins-hbase4.apache.org,38247,1677560224887 in 180 msec 2023-02-28 04:57:31,442 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=209, resume processing ppid=208 2023-02-28 04:57:31,442 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=203, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, ASSIGN in 342 msec 2023-02-28 04:57:31,442 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=209, ppid=208, state=SUCCESS; OpenRegionProcedure 35c83130614b8300868b867ad906a41e, server=jenkins-hbase4.apache.org,46629,1677560224813 in 183 msec 2023-02-28 04:57:31,442 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=214, resume processing ppid=206 2023-02-28 04:57:31,442 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=214, ppid=206, state=SUCCESS; OpenRegionProcedure 640ee1a6dbf01ebb011a298d972247c6, server=jenkins-hbase4.apache.org,36597,1677560224857 in 179 msec 2023-02-28 04:57:31,442 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=208, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, ASSIGN in 345 msec 2023-02-28 04:57:31,443 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=206, resume processing ppid=202 2023-02-28 04:57:31,443 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=206, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, ASSIGN in 345 msec 2023-02-28 04:57:31,444 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1677560251009 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-28 04:57:31,444 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560251444"}]},"ts":"1677560251444"} 2023-02-28 04:57:31,445 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1677560251009, state=ENABLED in hbase:meta 2023-02-28 04:57:31,452 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1677560251009 execute state=CREATE_TABLE_POST_OPERATION 2023-02-28 04:57:31,453 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=202, state=SUCCESS; CreateTableProcedure table=testCorruptedSnapshot-1677560251009 in 441 msec 2023-02-28 04:57:31,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-28 04:57:31,767 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCorruptedSnapshot-1677560251009, procId: 202 completed 2023-02-28 04:57:31,769 DEBUG [Listener at localhost/36897] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1677560251009 2023-02-28 04:57:31,769 DEBUG [Listener at localhost/36897] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:31,772 INFO [Listener at localhost/36897] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-28 04:57:31,785 INFO [Listener at localhost/36897] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1677560251009 2023-02-28 04:57:31,786 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1677560251009 2023-02-28 04:57:31,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=215, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1677560251009 2023-02-28 04:57:31,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-28 04:57:31,790 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560251790"}]},"ts":"1677560251790"} 2023-02-28 04:57:31,792 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1677560251009, state=DISABLING in hbase:meta 2023-02-28 04:57:31,795 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1677560251009 to state=DISABLING 2023-02-28 04:57:31,795 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, UNASSIGN}, {pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, UNASSIGN}, {pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, UNASSIGN}, {pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, UNASSIGN}, {pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, UNASSIGN}, {pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, UNASSIGN}] 2023-02-28 04:57:31,797 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, UNASSIGN 2023-02-28 04:57:31,797 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, UNASSIGN 2023-02-28 04:57:31,797 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, UNASSIGN 2023-02-28 04:57:31,797 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, UNASSIGN 2023-02-28 04:57:31,797 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, UNASSIGN 2023-02-28 04:57:31,797 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=9bc02e4052125a0d5af36b95e5b25a81, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:31,798 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560251797"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560251797"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560251797"}]},"ts":"1677560251797"} 2023-02-28 04:57:31,798 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=6d657e535457f1d48270505f48f3ceda, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:31,798 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=830ed62397c5222f09d35b7149e1b7db, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:31,798 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251798"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560251798"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560251798"}]},"ts":"1677560251798"} 2023-02-28 04:57:31,798 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=35c83130614b8300868b867ad906a41e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:31,798 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=640ee1a6dbf01ebb011a298d972247c6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:31,798 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560251798"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560251798"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560251798"}]},"ts":"1677560251798"} 2023-02-28 04:57:31,798 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251798"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560251798"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560251798"}]},"ts":"1677560251798"} 2023-02-28 04:57:31,798 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251798"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560251798"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560251798"}]},"ts":"1677560251798"} 2023-02-28 04:57:31,799 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=222, ppid=216, state=RUNNABLE; CloseRegionProcedure 9bc02e4052125a0d5af36b95e5b25a81, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:31,800 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=223, ppid=220, state=RUNNABLE; CloseRegionProcedure 6d657e535457f1d48270505f48f3ceda, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:31,800 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=224, ppid=221, state=RUNNABLE; CloseRegionProcedure 35c83130614b8300868b867ad906a41e, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:31,801 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=225, ppid=219, state=RUNNABLE; CloseRegionProcedure 640ee1a6dbf01ebb011a298d972247c6, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:31,802 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=226, ppid=217, state=RUNNABLE; CloseRegionProcedure 830ed62397c5222f09d35b7149e1b7db, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:31,803 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, UNASSIGN 2023-02-28 04:57:31,803 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=d67d3cb3ce05e60ff75f17313b64811d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:31,804 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251803"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560251803"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560251803"}]},"ts":"1677560251803"} 2023-02-28 04:57:31,805 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=227, ppid=218, state=RUNNABLE; CloseRegionProcedure d67d3cb3ce05e60ff75f17313b64811d, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:31,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:31,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 6d657e535457f1d48270505f48f3ceda, disabling compactions & flushes 2023-02-28 04:57:31,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:31,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:31,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. after waiting 0 ms 2023-02-28 04:57:31,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:31,952 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:31,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 35c83130614b8300868b867ad906a41e, disabling compactions & flushes 2023-02-28 04:57:31,953 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:31,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:31,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. after waiting 0 ms 2023-02-28 04:57:31,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:31,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:31,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 830ed62397c5222f09d35b7149e1b7db, disabling compactions & flushes 2023-02-28 04:57:31,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:31,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:31,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. after waiting 0 ms 2023-02-28 04:57:31,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:31,955 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-28 04:57:31,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-28 04:57:31,956 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:31,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 6d657e535457f1d48270505f48f3ceda: 2023-02-28 04:57:31,956 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:31,956 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 35c83130614b8300868b867ad906a41e: 2023-02-28 04:57:31,957 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-28 04:57:31,957 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:31,957 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 830ed62397c5222f09d35b7149e1b7db: 2023-02-28 04:57:31,957 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:31,957 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:31,957 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9bc02e4052125a0d5af36b95e5b25a81, disabling compactions & flushes 2023-02-28 04:57:31,957 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:31,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:31,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. after waiting 0 ms 2023-02-28 04:57:31,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:31,958 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=6d657e535457f1d48270505f48f3ceda, regionState=CLOSED 2023-02-28 04:57:31,958 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251958"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560251958"}]},"ts":"1677560251958"} 2023-02-28 04:57:31,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:31,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:31,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d67d3cb3ce05e60ff75f17313b64811d, disabling compactions & flushes 2023-02-28 04:57:31,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:31,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:31,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. after waiting 0 ms 2023-02-28 04:57:31,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:31,959 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=35c83130614b8300868b867ad906a41e, regionState=CLOSED 2023-02-28 04:57:31,959 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560251959"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560251959"}]},"ts":"1677560251959"} 2023-02-28 04:57:31,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:31,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:31,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 640ee1a6dbf01ebb011a298d972247c6, disabling compactions & flushes 2023-02-28 04:57:31,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:31,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:31,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. after waiting 0 ms 2023-02-28 04:57:31,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:31,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-28 04:57:31,961 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=830ed62397c5222f09d35b7149e1b7db, regionState=CLOSED 2023-02-28 04:57:31,962 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251961"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560251961"}]},"ts":"1677560251961"} 2023-02-28 04:57:31,963 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=223, resume processing ppid=220 2023-02-28 04:57:31,963 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=223, ppid=220, state=SUCCESS; CloseRegionProcedure 6d657e535457f1d48270505f48f3ceda, server=jenkins-hbase4.apache.org,38247,1677560224887 in 161 msec 2023-02-28 04:57:31,964 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=224, resume processing ppid=221 2023-02-28 04:57:31,964 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=224, ppid=221, state=SUCCESS; CloseRegionProcedure 35c83130614b8300868b867ad906a41e, server=jenkins-hbase4.apache.org,46629,1677560224813 in 162 msec 2023-02-28 04:57:31,964 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=220, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, UNASSIGN in 168 msec 2023-02-28 04:57:31,965 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=221, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, UNASSIGN in 169 msec 2023-02-28 04:57:31,966 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:31,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9bc02e4052125a0d5af36b95e5b25a81: 2023-02-28 04:57:31,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-28 04:57:31,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-28 04:57:31,968 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:31,968 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:31,968 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:31,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d67d3cb3ce05e60ff75f17313b64811d: 2023-02-28 04:57:31,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 640ee1a6dbf01ebb011a298d972247c6: 2023-02-28 04:57:31,968 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=9bc02e4052125a0d5af36b95e5b25a81, regionState=CLOSED 2023-02-28 04:57:31,969 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560251968"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560251968"}]},"ts":"1677560251968"} 2023-02-28 04:57:31,969 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=226, resume processing ppid=217 2023-02-28 04:57:31,969 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=226, ppid=217, state=SUCCESS; CloseRegionProcedure 830ed62397c5222f09d35b7149e1b7db, server=jenkins-hbase4.apache.org,36597,1677560224857 in 165 msec 2023-02-28 04:57:31,969 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:31,970 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=217, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, UNASSIGN in 174 msec 2023-02-28 04:57:31,970 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=d67d3cb3ce05e60ff75f17313b64811d, regionState=CLOSED 2023-02-28 04:57:31,970 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:31,970 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251970"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560251970"}]},"ts":"1677560251970"} 2023-02-28 04:57:31,970 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=640ee1a6dbf01ebb011a298d972247c6, regionState=CLOSED 2023-02-28 04:57:31,971 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560251970"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560251970"}]},"ts":"1677560251970"} 2023-02-28 04:57:31,971 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=222, resume processing ppid=216 2023-02-28 04:57:31,972 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=222, ppid=216, state=SUCCESS; CloseRegionProcedure 9bc02e4052125a0d5af36b95e5b25a81, server=jenkins-hbase4.apache.org,38247,1677560224887 in 171 msec 2023-02-28 04:57:31,973 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=216, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, UNASSIGN in 176 msec 2023-02-28 04:57:31,974 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=227, resume processing ppid=218 2023-02-28 04:57:31,974 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=227, ppid=218, state=SUCCESS; CloseRegionProcedure d67d3cb3ce05e60ff75f17313b64811d, server=jenkins-hbase4.apache.org,46629,1677560224813 in 166 msec 2023-02-28 04:57:31,974 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=225, resume processing ppid=219 2023-02-28 04:57:31,974 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=225, ppid=219, state=SUCCESS; CloseRegionProcedure 640ee1a6dbf01ebb011a298d972247c6, server=jenkins-hbase4.apache.org,36597,1677560224857 in 171 msec 2023-02-28 04:57:31,975 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=218, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, UNASSIGN in 179 msec 2023-02-28 04:57:31,975 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=219, resume processing ppid=215 2023-02-28 04:57:31,975 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=219, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, UNASSIGN in 179 msec 2023-02-28 04:57:31,976 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560251976"}]},"ts":"1677560251976"} 2023-02-28 04:57:31,977 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1677560251009, state=DISABLED in hbase:meta 2023-02-28 04:57:31,979 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1677560251009 to state=DISABLED 2023-02-28 04:57:31,980 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=215, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1677560251009 in 193 msec 2023-02-28 04:57:32,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-28 04:57:32,040 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1677560251009, procId: 215 completed 2023-02-28 04:57:32,041 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1677560251009 table=testCorruptedSnapshot-1677560251009 type=FLUSH ttl=0 } 2023-02-28 04:57:32,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1677560252041 (current time:1677560252041). 2023-02-28 04:57:32,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-28 04:57:32,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] zookeeper.ReadOnlyZKClient(139): Connect 0x3f95ad71 to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:32,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6d243d6b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:32,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:32,049 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41628, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:32,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f95ad71 to 127.0.0.1:54355 2023-02-28 04:57:32,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:32,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-28 04:57:32,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1677560251009 table=testCorruptedSnapshot-1677560251009 type=FLUSH ttl=0 } 2023-02-28 04:57:32,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1677560251009, type=EXCLUSIVE 2023-02-28 04:57:32,053 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1677560251009, type=EXCLUSIVE 2023-02-28 04:57:32,054 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=228, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1677560251009, type=EXCLUSIVE; timeout=600000, timestamp=1677560852054 2023-02-28 04:57:32,054 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1677560251009 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1677560251009 2023-02-28 04:57:32,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1677560251009 table=testCorruptedSnapshot-1677560251009 type=FLUSH ttl=0 } 2023-02-28 04:57:32,054 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1677560251009 table=testCorruptedSnapshot-1677560251009 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-28 04:57:32,054 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-28 04:57:32,065 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1677560251009 table=testCorruptedSnapshot-1677560251009 type=DISABLED ttl=0 } 2023-02-28 04:57:32,066 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:32,066 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:32,067 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:32,067 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:32,067 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:32,067 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:32,067 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:32,067 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:32,067 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:32,067 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:32,067 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:32,067 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:32,068 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:32,068 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:32,068 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:32,069 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:32,069 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:32,069 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:32,069 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:32,069 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:32,070 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-28 04:57:32,095 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-28 04:57:32,304 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-28 04:57:32,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1677560251009 table=testCorruptedSnapshot-1677560251009 type=FLUSH ttl=0 } is done 2023-02-28 04:57:32,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(404): Snapshoting '{ ss=emptySnaptb-1677560251009 table=testCorruptedSnapshot-1677560251009 type=FLUSH ttl=0 }' is still in progress! 2023-02-28 04:57:32,305 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-28 04:57:32,551 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1677560251009 table=testCorruptedSnapshot-1677560251009 type=DISABLED ttl=0 } as finished. 2023-02-28 04:57:32,551 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1677560251009 2023-02-28 04:57:32,552 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/emptySnaptb-1677560251009 2023-02-28 04:57:32,568 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/emptySnaptb-1677560251009 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/emptySnaptb-1677560251009 2023-02-28 04:57:32,583 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1677560251009 of table testCorruptedSnapshot-1677560251009 completed 2023-02-28 04:57:32,583 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/emptySnaptb-1677560251009 2023-02-28 04:57:32,583 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/emptySnaptb-1677560251009 2023-02-28 04:57:32,583 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1677560251009 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1677560251009 at 1677560252054Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1677560251009 table=testCorruptedSnapshot-1677560251009 type=DISABLED ttl=0 } at 1677560252065 (+11 ms)Consolidate snapshot: emptySnaptb-1677560251009 at 1677560252551 (+486 ms)Loading Region manifests for emptySnaptb-1677560251009 at 1677560252552 (+1 ms)Writing data manifest for emptySnaptb-1677560251009 at 1677560252555 (+3 ms)Verifying snapshot: emptySnaptb-1677560251009 at 1677560252563 (+8 ms)Snapshot emptySnaptb-1677560251009 of table testCorruptedSnapshot-1677560251009 completed at 1677560252583 (+20 ms) 2023-02-28 04:57:32,585 DEBUG [PEWorker-3] locking.LockProcedure(242): UNLOCKED pid=228, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1677560251009, type=EXCLUSIVE 2023-02-28 04:57:32,586 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=228, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1677560251009, type=EXCLUSIVE in 534 msec 2023-02-28 04:57:32,806 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-28 04:57:32,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1677560251009 table=testCorruptedSnapshot-1677560251009 type=FLUSH ttl=0 } is done 2023-02-28 04:57:32,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1677560251009 table=testCorruptedSnapshot-1677560251009 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-28 04:57:32,807 INFO [Listener at localhost/36897] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1677560251009 2023-02-28 04:57:32,807 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1677560251009 2023-02-28 04:57:32,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=229, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1677560251009 2023-02-28 04:57:32,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-28 04:57:32,810 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1677560251009 2023-02-28 04:57:32,811 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560252810"}]},"ts":"1677560252810"} 2023-02-28 04:57:32,812 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1677560251009, state=ENABLING in hbase:meta 2023-02-28 04:57:32,815 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-28 04:57:32,815 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, ASSIGN}, {pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, ASSIGN}, {pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, ASSIGN}, {pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, ASSIGN}, {pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, ASSIGN}, {pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, ASSIGN}] 2023-02-28 04:57:32,816 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, ASSIGN 2023-02-28 04:57:32,816 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, ASSIGN 2023-02-28 04:57:32,816 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, ASSIGN 2023-02-28 04:57:32,817 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, ASSIGN 2023-02-28 04:57:32,817 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:32,817 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:32,817 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,36597,1677560224857; forceNewPlan=false, retain=true 2023-02-28 04:57:32,817 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,38247,1677560224887; forceNewPlan=false, retain=true 2023-02-28 04:57:32,817 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, ASSIGN 2023-02-28 04:57:32,817 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:32,817 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:32,817 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,38247,1677560224887; forceNewPlan=false, retain=true 2023-02-28 04:57:32,818 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46629,1677560224813; forceNewPlan=false, retain=true 2023-02-28 04:57:32,818 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, ASSIGN 2023-02-28 04:57:32,818 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:32,818 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46629,1677560224813; forceNewPlan=false, retain=true 2023-02-28 04:57:32,819 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:32,819 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,36597,1677560224857; forceNewPlan=false, retain=true 2023-02-28 04:57:32,967 INFO [jenkins-hbase4:42505] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-28 04:57:32,968 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=9bc02e4052125a0d5af36b95e5b25a81, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:32,968 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=6d657e535457f1d48270505f48f3ceda, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:32,968 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=830ed62397c5222f09d35b7149e1b7db, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:32,968 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=35c83130614b8300868b867ad906a41e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:32,968 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560252968"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560252968"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560252968"}]},"ts":"1677560252968"} 2023-02-28 04:57:32,968 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560252968"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560252968"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560252968"}]},"ts":"1677560252968"} 2023-02-28 04:57:32,968 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=d67d3cb3ce05e60ff75f17313b64811d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:32,968 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560252968"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560252968"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560252968"}]},"ts":"1677560252968"} 2023-02-28 04:57:32,968 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560252968"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560252968"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560252968"}]},"ts":"1677560252968"} 2023-02-28 04:57:32,968 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560252968"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560252968"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560252968"}]},"ts":"1677560252968"} 2023-02-28 04:57:32,970 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=236, ppid=231, state=RUNNABLE; OpenRegionProcedure 830ed62397c5222f09d35b7149e1b7db, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:32,970 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=237, ppid=235, state=RUNNABLE; OpenRegionProcedure 35c83130614b8300868b867ad906a41e, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:32,971 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=238, ppid=234, state=RUNNABLE; OpenRegionProcedure 6d657e535457f1d48270505f48f3ceda, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:32,971 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=239, ppid=232, state=RUNNABLE; OpenRegionProcedure d67d3cb3ce05e60ff75f17313b64811d, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:32,972 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=240, ppid=230, state=RUNNABLE; OpenRegionProcedure 9bc02e4052125a0d5af36b95e5b25a81, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:32,972 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=640ee1a6dbf01ebb011a298d972247c6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:32,972 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560252972"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560252972"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560252972"}]},"ts":"1677560252972"} 2023-02-28 04:57:32,979 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=241, ppid=233, state=RUNNABLE; OpenRegionProcedure 640ee1a6dbf01ebb011a298d972247c6, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:33,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-28 04:57:33,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:33,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 640ee1a6dbf01ebb011a298d972247c6, NAME => 'testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.', STARTKEY => '3', ENDKEY => '4'} 2023-02-28 04:57:33,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:33,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:33,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:33,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:33,124 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:33,124 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:33,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9bc02e4052125a0d5af36b95e5b25a81, NAME => 'testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.', STARTKEY => '', ENDKEY => '1'} 2023-02-28 04:57:33,124 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 35c83130614b8300868b867ad906a41e, NAME => 'testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.', STARTKEY => '5', ENDKEY => ''} 2023-02-28 04:57:33,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:33,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:33,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:33,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:33,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:33,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:33,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:33,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:33,126 INFO [StoreOpener-640ee1a6dbf01ebb011a298d972247c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:33,126 INFO [StoreOpener-9bc02e4052125a0d5af36b95e5b25a81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:33,126 INFO [StoreOpener-35c83130614b8300868b867ad906a41e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:33,127 DEBUG [StoreOpener-9bc02e4052125a0d5af36b95e5b25a81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf 2023-02-28 04:57:33,127 DEBUG [StoreOpener-9bc02e4052125a0d5af36b95e5b25a81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf 2023-02-28 04:57:33,127 DEBUG [StoreOpener-640ee1a6dbf01ebb011a298d972247c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf 2023-02-28 04:57:33,127 DEBUG [StoreOpener-35c83130614b8300868b867ad906a41e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf 2023-02-28 04:57:33,127 DEBUG [StoreOpener-640ee1a6dbf01ebb011a298d972247c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf 2023-02-28 04:57:33,127 DEBUG [StoreOpener-35c83130614b8300868b867ad906a41e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf 2023-02-28 04:57:33,127 INFO [StoreOpener-9bc02e4052125a0d5af36b95e5b25a81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9bc02e4052125a0d5af36b95e5b25a81 columnFamilyName cf 2023-02-28 04:57:33,127 INFO [StoreOpener-35c83130614b8300868b867ad906a41e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 35c83130614b8300868b867ad906a41e columnFamilyName cf 2023-02-28 04:57:33,127 INFO [StoreOpener-640ee1a6dbf01ebb011a298d972247c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 640ee1a6dbf01ebb011a298d972247c6 columnFamilyName cf 2023-02-28 04:57:33,128 INFO [StoreOpener-35c83130614b8300868b867ad906a41e-1] regionserver.HStore(310): Store=35c83130614b8300868b867ad906a41e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:33,128 INFO [StoreOpener-9bc02e4052125a0d5af36b95e5b25a81-1] regionserver.HStore(310): Store=9bc02e4052125a0d5af36b95e5b25a81/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:33,128 INFO [StoreOpener-640ee1a6dbf01ebb011a298d972247c6-1] regionserver.HStore(310): Store=640ee1a6dbf01ebb011a298d972247c6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:33,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:33,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:33,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e 2023-02-28 04:57:33,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:33,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e 2023-02-28 04:57:33,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:33,132 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:33,132 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:33,132 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:33,133 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9bc02e4052125a0d5af36b95e5b25a81; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60539858, jitterRate=-0.09788581728935242}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:33,133 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9bc02e4052125a0d5af36b95e5b25a81: 2023-02-28 04:57:33,133 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 35c83130614b8300868b867ad906a41e; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66690607, jitterRate=-0.006232514977455139}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:33,133 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 35c83130614b8300868b867ad906a41e: 2023-02-28 04:57:33,133 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 640ee1a6dbf01ebb011a298d972247c6; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69694870, jitterRate=0.0385344922542572}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:33,133 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 640ee1a6dbf01ebb011a298d972247c6: 2023-02-28 04:57:33,134 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e., pid=237, masterSystemTime=1677560253122 2023-02-28 04:57:33,134 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81., pid=240, masterSystemTime=1677560253122 2023-02-28 04:57:33,134 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6., pid=241, masterSystemTime=1677560253121 2023-02-28 04:57:33,135 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:33,135 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:33,135 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:33,135 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d67d3cb3ce05e60ff75f17313b64811d, NAME => 'testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.', STARTKEY => '2', ENDKEY => '3'} 2023-02-28 04:57:33,135 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:33,135 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:33,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:33,136 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:33,136 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=35c83130614b8300868b867ad906a41e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:33,136 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560253136"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560253136"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560253136"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560253136"}]},"ts":"1677560253136"} 2023-02-28 04:57:33,136 INFO [StoreOpener-d67d3cb3ce05e60ff75f17313b64811d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:33,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:33,137 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:33,137 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:33,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6d657e535457f1d48270505f48f3ceda, NAME => 'testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.', STARTKEY => '4', ENDKEY => '5'} 2023-02-28 04:57:33,137 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=9bc02e4052125a0d5af36b95e5b25a81, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:33,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:33,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:33,137 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560253137"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560253137"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560253137"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560253137"}]},"ts":"1677560253137"} 2023-02-28 04:57:33,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:33,137 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:33,137 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:33,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 830ed62397c5222f09d35b7149e1b7db, NAME => 'testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.', STARTKEY => '1', ENDKEY => '2'} 2023-02-28 04:57:33,138 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:33,138 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:33,138 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:33,138 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:33,137 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:33,137 DEBUG [StoreOpener-d67d3cb3ce05e60ff75f17313b64811d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf 2023-02-28 04:57:33,138 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:33,138 DEBUG [StoreOpener-d67d3cb3ce05e60ff75f17313b64811d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf 2023-02-28 04:57:33,139 INFO [StoreOpener-d67d3cb3ce05e60ff75f17313b64811d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d67d3cb3ce05e60ff75f17313b64811d columnFamilyName cf 2023-02-28 04:57:33,139 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=640ee1a6dbf01ebb011a298d972247c6, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:33,139 INFO [StoreOpener-830ed62397c5222f09d35b7149e1b7db-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:33,139 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560253139"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560253139"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560253139"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560253139"}]},"ts":"1677560253139"} 2023-02-28 04:57:33,139 INFO [StoreOpener-6d657e535457f1d48270505f48f3ceda-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:33,140 INFO [StoreOpener-d67d3cb3ce05e60ff75f17313b64811d-1] regionserver.HStore(310): Store=d67d3cb3ce05e60ff75f17313b64811d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:33,140 DEBUG [StoreOpener-830ed62397c5222f09d35b7149e1b7db-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf 2023-02-28 04:57:33,140 DEBUG [StoreOpener-830ed62397c5222f09d35b7149e1b7db-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf 2023-02-28 04:57:33,140 DEBUG [StoreOpener-6d657e535457f1d48270505f48f3ceda-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf 2023-02-28 04:57:33,140 DEBUG [StoreOpener-6d657e535457f1d48270505f48f3ceda-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf 2023-02-28 04:57:33,140 INFO [StoreOpener-830ed62397c5222f09d35b7149e1b7db-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 830ed62397c5222f09d35b7149e1b7db columnFamilyName cf 2023-02-28 04:57:33,141 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:33,141 INFO [StoreOpener-6d657e535457f1d48270505f48f3ceda-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6d657e535457f1d48270505f48f3ceda columnFamilyName cf 2023-02-28 04:57:33,142 INFO [StoreOpener-830ed62397c5222f09d35b7149e1b7db-1] regionserver.HStore(310): Store=830ed62397c5222f09d35b7149e1b7db/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:33,142 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:33,142 INFO [StoreOpener-6d657e535457f1d48270505f48f3ceda-1] regionserver.HStore(310): Store=6d657e535457f1d48270505f48f3ceda/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:33,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:33,143 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=237, resume processing ppid=235 2023-02-28 04:57:33,143 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=237, ppid=235, state=SUCCESS; OpenRegionProcedure 35c83130614b8300868b867ad906a41e, server=jenkins-hbase4.apache.org,46629,1677560224813 in 169 msec 2023-02-28 04:57:33,143 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=240, resume processing ppid=230 2023-02-28 04:57:33,143 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=240, ppid=230, state=SUCCESS; OpenRegionProcedure 9bc02e4052125a0d5af36b95e5b25a81, server=jenkins-hbase4.apache.org,38247,1677560224887 in 168 msec 2023-02-28 04:57:33,143 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:33,143 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=241, resume processing ppid=233 2023-02-28 04:57:33,144 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=241, ppid=233, state=SUCCESS; OpenRegionProcedure 640ee1a6dbf01ebb011a298d972247c6, server=jenkins-hbase4.apache.org,36597,1677560224857 in 162 msec 2023-02-28 04:57:33,144 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:33,144 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=235, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, ASSIGN in 328 msec 2023-02-28 04:57:33,145 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:33,145 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=230, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, ASSIGN in 328 msec 2023-02-28 04:57:33,145 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=233, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, ASSIGN in 328 msec 2023-02-28 04:57:33,146 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:33,146 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d67d3cb3ce05e60ff75f17313b64811d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65185771, jitterRate=-0.02865631878376007}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:33,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d67d3cb3ce05e60ff75f17313b64811d: 2023-02-28 04:57:33,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:33,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:33,147 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d., pid=239, masterSystemTime=1677560253122 2023-02-28 04:57:33,147 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 6d657e535457f1d48270505f48f3ceda; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62125373, jitterRate=-0.07425980269908905}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:33,147 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 830ed62397c5222f09d35b7149e1b7db; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59553592, jitterRate=-0.11258232593536377}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:33,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 6d657e535457f1d48270505f48f3ceda: 2023-02-28 04:57:33,147 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 830ed62397c5222f09d35b7149e1b7db: 2023-02-28 04:57:33,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db., pid=236, masterSystemTime=1677560253121 2023-02-28 04:57:33,148 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda., pid=238, masterSystemTime=1677560253122 2023-02-28 04:57:33,149 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:33,149 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:33,149 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=d67d3cb3ce05e60ff75f17313b64811d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:33,149 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560253149"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560253149"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560253149"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560253149"}]},"ts":"1677560253149"} 2023-02-28 04:57:33,149 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:33,149 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:33,150 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=830ed62397c5222f09d35b7149e1b7db, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:33,150 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560253150"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560253150"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560253150"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560253150"}]},"ts":"1677560253150"} 2023-02-28 04:57:33,150 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=6d657e535457f1d48270505f48f3ceda, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:33,150 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560253150"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560253150"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560253150"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560253150"}]},"ts":"1677560253150"} 2023-02-28 04:57:33,153 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=239, resume processing ppid=232 2023-02-28 04:57:33,153 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=239, ppid=232, state=SUCCESS; OpenRegionProcedure d67d3cb3ce05e60ff75f17313b64811d, server=jenkins-hbase4.apache.org,46629,1677560224813 in 180 msec 2023-02-28 04:57:33,153 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=236, resume processing ppid=231 2023-02-28 04:57:33,153 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=236, ppid=231, state=SUCCESS; OpenRegionProcedure 830ed62397c5222f09d35b7149e1b7db, server=jenkins-hbase4.apache.org,36597,1677560224857 in 182 msec 2023-02-28 04:57:33,154 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=238, resume processing ppid=234 2023-02-28 04:57:33,154 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=232, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, ASSIGN in 338 msec 2023-02-28 04:57:33,154 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=231, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, ASSIGN in 338 msec 2023-02-28 04:57:33,154 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=238, ppid=234, state=SUCCESS; OpenRegionProcedure 6d657e535457f1d48270505f48f3ceda, server=jenkins-hbase4.apache.org,38247,1677560224887 in 181 msec 2023-02-28 04:57:33,155 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=234, resume processing ppid=229 2023-02-28 04:57:33,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:33,156 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=234, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, ASSIGN in 339 msec 2023-02-28 04:57:33,156 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:33,156 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560253156"}]},"ts":"1677560253156"} 2023-02-28 04:57:33,157 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1677560251009, state=ENABLED in hbase:meta 2023-02-28 04:57:33,159 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1677560251009' was successfully enabled. 2023-02-28 04:57:33,160 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=229, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1677560251009 in 352 msec 2023-02-28 04:57:33,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-28 04:57:33,562 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1677560251009, procId: 229 completed 2023-02-28 04:57:33,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:33,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:33,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:33,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:33,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:33,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:33,577 DEBUG [Listener at localhost/36897] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1677560251009 2023-02-28 04:57:33,577 DEBUG [Listener at localhost/36897] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:33,577 INFO [Listener at localhost/36897] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-28 04:57:33,588 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-28 04:57:33,589 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-28 04:57:33,590 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-28 04:57:33,591 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-28 04:57:33,592 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-28 04:57:33,595 INFO [Listener at localhost/36897] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1677560251009 2023-02-28 04:57:33,596 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1677560251009 2023-02-28 04:57:33,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=242, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1677560251009 2023-02-28 04:57:33,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-28 04:57:33,599 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560253599"}]},"ts":"1677560253599"} 2023-02-28 04:57:33,601 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1677560251009, state=DISABLING in hbase:meta 2023-02-28 04:57:33,603 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1677560251009 to state=DISABLING 2023-02-28 04:57:33,603 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, UNASSIGN}, {pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, UNASSIGN}, {pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, UNASSIGN}, {pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, UNASSIGN}, {pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, UNASSIGN}, {pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, UNASSIGN}] 2023-02-28 04:57:33,606 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, UNASSIGN 2023-02-28 04:57:33,606 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, UNASSIGN 2023-02-28 04:57:33,606 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, UNASSIGN 2023-02-28 04:57:33,606 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, UNASSIGN 2023-02-28 04:57:33,606 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, UNASSIGN 2023-02-28 04:57:33,607 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=d67d3cb3ce05e60ff75f17313b64811d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:33,607 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=9bc02e4052125a0d5af36b95e5b25a81, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:33,607 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560253607"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560253607"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560253607"}]},"ts":"1677560253607"} 2023-02-28 04:57:33,607 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560253607"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560253607"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560253607"}]},"ts":"1677560253607"} 2023-02-28 04:57:33,607 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=830ed62397c5222f09d35b7149e1b7db, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:33,607 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=640ee1a6dbf01ebb011a298d972247c6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:33,607 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560253607"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560253607"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560253607"}]},"ts":"1677560253607"} 2023-02-28 04:57:33,607 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560253607"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560253607"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560253607"}]},"ts":"1677560253607"} 2023-02-28 04:57:33,607 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=35c83130614b8300868b867ad906a41e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:33,608 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560253607"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560253607"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560253607"}]},"ts":"1677560253607"} 2023-02-28 04:57:33,608 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=249, ppid=243, state=RUNNABLE; CloseRegionProcedure 9bc02e4052125a0d5af36b95e5b25a81, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:33,609 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=250, ppid=245, state=RUNNABLE; CloseRegionProcedure d67d3cb3ce05e60ff75f17313b64811d, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:33,610 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=251, ppid=244, state=RUNNABLE; CloseRegionProcedure 830ed62397c5222f09d35b7149e1b7db, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:33,611 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=252, ppid=246, state=RUNNABLE; CloseRegionProcedure 640ee1a6dbf01ebb011a298d972247c6, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:33,612 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, UNASSIGN 2023-02-28 04:57:33,612 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=253, ppid=248, state=RUNNABLE; CloseRegionProcedure 35c83130614b8300868b867ad906a41e, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:33,613 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=6d657e535457f1d48270505f48f3ceda, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:33,613 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560253613"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560253613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560253613"}]},"ts":"1677560253613"} 2023-02-28 04:57:33,615 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=254, ppid=247, state=RUNNABLE; CloseRegionProcedure 6d657e535457f1d48270505f48f3ceda, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:33,761 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:33,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 6d657e535457f1d48270505f48f3ceda, disabling compactions & flushes 2023-02-28 04:57:33,761 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:33,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:33,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. after waiting 0 ms 2023-02-28 04:57:33,761 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:33,761 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 6d657e535457f1d48270505f48f3ceda 1/1 column families, dataSize=1.63 KB heapSize=3.77 KB 2023-02-28 04:57:33,762 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:33,762 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 35c83130614b8300868b867ad906a41e, disabling compactions & flushes 2023-02-28 04:57:33,762 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:33,762 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:33,762 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. after waiting 0 ms 2023-02-28 04:57:33,762 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:33,763 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 35c83130614b8300868b867ad906a41e 1/1 column families, dataSize=22.90 KB heapSize=49.47 KB 2023-02-28 04:57:33,763 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:33,763 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 640ee1a6dbf01ebb011a298d972247c6, disabling compactions & flushes 2023-02-28 04:57:33,763 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:33,763 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:33,764 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. after waiting 0 ms 2023-02-28 04:57:33,764 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:33,765 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 640ee1a6dbf01ebb011a298d972247c6 1/1 column families, dataSize=1.90 KB heapSize=4.33 KB 2023-02-28 04:57:33,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302285080540a74f24d9296085cd67cf2022f to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/a87ff679a2f3e71d9181a67b7542122c202302285080540a74f24d9296085cd67cf2022f 2023-02-28 04:57:33,786 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf3202302284ee665ca2c9e4e978c48fdb107b930b6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302284ee665ca2c9e4e978c48fdb107b930b6 2023-02-28 04:57:33,787 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230228cb27ffebb03648b289747a1cd18ac8e8 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/e4da3b7fbbce2345d7772b0674a318d520230228cb27ffebb03648b289747a1cd18ac8e8 2023-02-28 04:57:33,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/.tmp/cf/d9d1454edff6491c896e22be29002893 2023-02-28 04:57:33,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/.tmp/cf/85b0830a8ec24e8e9d58e18a321b544e 2023-02-28 04:57:33,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/.tmp/cf/648834311b524feb8715995350800fa8 2023-02-28 04:57:33,800 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/.tmp/cf/d9d1454edff6491c896e22be29002893 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf/d9d1454edff6491c896e22be29002893 2023-02-28 04:57:33,801 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/.tmp/cf/648834311b524feb8715995350800fa8 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf/648834311b524feb8715995350800fa8 2023-02-28 04:57:33,801 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/.tmp/cf/85b0830a8ec24e8e9d58e18a321b544e as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf/85b0830a8ec24e8e9d58e18a321b544e 2023-02-28 04:57:33,803 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf/d9d1454edff6491c896e22be29002893, entries=29, sequenceid=8, filesize=9.9 K 2023-02-28 04:57:33,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf/648834311b524feb8715995350800fa8, entries=25, sequenceid=8, filesize=9.2 K 2023-02-28 04:57:33,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.90 KB/1941, heapSize ~4.31 KB/4416, currentSize=0 B/0 for 640ee1a6dbf01ebb011a298d972247c6 in 40ms, sequenceid=8, compaction requested=false 2023-02-28 04:57:33,804 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1677560251009' 2023-02-28 04:57:33,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.63 KB/1673, heapSize ~3.75 KB/3840, currentSize=0 B/0 for 6d657e535457f1d48270505f48f3ceda in 43ms, sequenceid=8, compaction requested=false 2023-02-28 04:57:33,804 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1677560251009' 2023-02-28 04:57:33,806 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf/85b0830a8ec24e8e9d58e18a321b544e, entries=350, sequenceid=8, filesize=65.2 K 2023-02-28 04:57:33,807 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.90 KB/23446, heapSize ~49.45 KB/50640, currentSize=0 B/0 for 35c83130614b8300868b867ad906a41e in 44ms, sequenceid=8, compaction requested=false 2023-02-28 04:57:33,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-28 04:57:33,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-28 04:57:33,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-28 04:57:33,810 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:33,810 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:33,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 6d657e535457f1d48270505f48f3ceda: 2023-02-28 04:57:33,810 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:33,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 35c83130614b8300868b867ad906a41e: 2023-02-28 04:57:33,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 640ee1a6dbf01ebb011a298d972247c6: 2023-02-28 04:57:33,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:33,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:33,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d67d3cb3ce05e60ff75f17313b64811d, disabling compactions & flushes 2023-02-28 04:57:33,812 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=35c83130614b8300868b867ad906a41e, regionState=CLOSED 2023-02-28 04:57:33,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:33,812 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560253812"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560253812"}]},"ts":"1677560253812"} 2023-02-28 04:57:33,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:33,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:33,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:33,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. after waiting 0 ms 2023-02-28 04:57:33,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9bc02e4052125a0d5af36b95e5b25a81, disabling compactions & flushes 2023-02-28 04:57:33,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:33,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:33,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:33,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing d67d3cb3ce05e60ff75f17313b64811d 1/1 column families, dataSize=2.62 KB heapSize=5.88 KB 2023-02-28 04:57:33,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. after waiting 0 ms 2023-02-28 04:57:33,812 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:33,812 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 9bc02e4052125a0d5af36b95e5b25a81 1/1 column families, dataSize=1.44 KB heapSize=3.34 KB 2023-02-28 04:57:33,813 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=6d657e535457f1d48270505f48f3ceda, regionState=CLOSED 2023-02-28 04:57:33,813 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560253813"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560253813"}]},"ts":"1677560253813"} 2023-02-28 04:57:33,814 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:33,814 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:33,814 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 830ed62397c5222f09d35b7149e1b7db, disabling compactions & flushes 2023-02-28 04:57:33,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:33,815 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:33,815 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. after waiting 0 ms 2023-02-28 04:57:33,815 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:33,815 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 830ed62397c5222f09d35b7149e1b7db 1/1 column families, dataSize=2.22 KB heapSize=5.03 KB 2023-02-28 04:57:33,815 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=640ee1a6dbf01ebb011a298d972247c6, regionState=CLOSED 2023-02-28 04:57:33,815 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560253815"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560253815"}]},"ts":"1677560253815"} 2023-02-28 04:57:33,819 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=253, resume processing ppid=248 2023-02-28 04:57:33,820 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=253, ppid=248, state=SUCCESS; CloseRegionProcedure 35c83130614b8300868b867ad906a41e, server=jenkins-hbase4.apache.org,46629,1677560224813 in 203 msec 2023-02-28 04:57:33,821 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=254, resume processing ppid=247 2023-02-28 04:57:33,821 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=254, ppid=247, state=SUCCESS; CloseRegionProcedure 6d657e535457f1d48270505f48f3ceda, server=jenkins-hbase4.apache.org,38247,1677560224887 in 203 msec 2023-02-28 04:57:33,822 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=252, resume processing ppid=246 2023-02-28 04:57:33,822 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=248, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, UNASSIGN in 217 msec 2023-02-28 04:57:33,822 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=252, ppid=246, state=SUCCESS; CloseRegionProcedure 640ee1a6dbf01ebb011a298d972247c6, server=jenkins-hbase4.apache.org,36597,1677560224857 in 208 msec 2023-02-28 04:57:33,822 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=247, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, UNASSIGN in 218 msec 2023-02-28 04:57:33,823 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=246, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, UNASSIGN in 219 msec 2023-02-28 04:57:33,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023022849be121c134747729f78fa30b34bf23d to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/d41d8cd98f00b204e9800998ecf8427e2023022849be121c134747729f78fa30b34bf23d 2023-02-28 04:57:33,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230228913dd8b6e2a34685aa1e0ea8bc127121 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/c4ca4238a0b923820dcc509a6f75849b20230228913dd8b6e2a34685aa1e0ea8bc127121 2023-02-28 04:57:33,841 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302283c19999ffdc44b829b28642a75d868ef to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/c81e728d9d4c2f636f067f89cc14862c202302283c19999ffdc44b829b28642a75d868ef 2023-02-28 04:57:33,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-28 04:57:33,855 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/.tmp/cf/a96fd165941e4d898d87f6cfa84457f8 2023-02-28 04:57:33,856 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/.tmp/cf/46c8c89265b5471ab4ddde49ca842fa6 2023-02-28 04:57:33,856 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/.tmp/cf/580c9ca6b54c4aa991c240159d3901be 2023-02-28 04:57:33,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/.tmp/cf/a96fd165941e4d898d87f6cfa84457f8 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf/a96fd165941e4d898d87f6cfa84457f8 2023-02-28 04:57:33,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/.tmp/cf/46c8c89265b5471ab4ddde49ca842fa6 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf/46c8c89265b5471ab4ddde49ca842fa6 2023-02-28 04:57:33,860 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/.tmp/cf/580c9ca6b54c4aa991c240159d3901be as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf/580c9ca6b54c4aa991c240159d3901be 2023-02-28 04:57:33,862 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf/46c8c89265b5471ab4ddde49ca842fa6, entries=34, sequenceid=8, filesize=10.8 K 2023-02-28 04:57:33,862 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf/a96fd165941e4d898d87f6cfa84457f8, entries=40, sequenceid=8, filesize=11.8 K 2023-02-28 04:57:33,862 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf/580c9ca6b54c4aa991c240159d3901be, entries=22, sequenceid=8, filesize=8.7 K 2023-02-28 04:57:33,863 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.22 KB/2276, heapSize ~5.02 KB/5136, currentSize=0 B/0 for 830ed62397c5222f09d35b7149e1b7db in 48ms, sequenceid=8, compaction requested=false 2023-02-28 04:57:33,863 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.44 KB/1472, heapSize ~3.33 KB/3408, currentSize=0 B/0 for 9bc02e4052125a0d5af36b95e5b25a81 in 51ms, sequenceid=8, compaction requested=false 2023-02-28 04:57:33,863 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.62 KB/2678, heapSize ~5.86 KB/6000, currentSize=0 B/0 for d67d3cb3ce05e60ff75f17313b64811d in 51ms, sequenceid=8, compaction requested=false 2023-02-28 04:57:33,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-28 04:57:33,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-28 04:57:33,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-28 04:57:33,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:33,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9bc02e4052125a0d5af36b95e5b25a81: 2023-02-28 04:57:33,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:33,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:33,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d67d3cb3ce05e60ff75f17313b64811d: 2023-02-28 04:57:33,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 830ed62397c5222f09d35b7149e1b7db: 2023-02-28 04:57:33,867 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:33,868 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=9bc02e4052125a0d5af36b95e5b25a81, regionState=CLOSED 2023-02-28 04:57:33,868 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560253868"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560253868"}]},"ts":"1677560253868"} 2023-02-28 04:57:33,868 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:33,868 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=d67d3cb3ce05e60ff75f17313b64811d, regionState=CLOSED 2023-02-28 04:57:33,868 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560253868"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560253868"}]},"ts":"1677560253868"} 2023-02-28 04:57:33,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:33,869 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=830ed62397c5222f09d35b7149e1b7db, regionState=CLOSED 2023-02-28 04:57:33,869 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560253869"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560253869"}]},"ts":"1677560253869"} 2023-02-28 04:57:33,871 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=249, resume processing ppid=243 2023-02-28 04:57:33,871 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=249, ppid=243, state=SUCCESS; CloseRegionProcedure 9bc02e4052125a0d5af36b95e5b25a81, server=jenkins-hbase4.apache.org,38247,1677560224887 in 261 msec 2023-02-28 04:57:33,871 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=250, resume processing ppid=245 2023-02-28 04:57:33,871 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=250, ppid=245, state=SUCCESS; CloseRegionProcedure d67d3cb3ce05e60ff75f17313b64811d, server=jenkins-hbase4.apache.org,46629,1677560224813 in 261 msec 2023-02-28 04:57:33,872 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=243, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, UNASSIGN in 268 msec 2023-02-28 04:57:33,872 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=251, resume processing ppid=244 2023-02-28 04:57:33,872 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=245, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, UNASSIGN in 268 msec 2023-02-28 04:57:33,872 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=251, ppid=244, state=SUCCESS; CloseRegionProcedure 830ed62397c5222f09d35b7149e1b7db, server=jenkins-hbase4.apache.org,36597,1677560224857 in 261 msec 2023-02-28 04:57:33,874 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=244, resume processing ppid=242 2023-02-28 04:57:33,874 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=244, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, UNASSIGN in 269 msec 2023-02-28 04:57:33,874 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560253874"}]},"ts":"1677560253874"} 2023-02-28 04:57:33,875 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1677560251009, state=DISABLED in hbase:meta 2023-02-28 04:57:33,877 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1677560251009 to state=DISABLED 2023-02-28 04:57:33,878 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=242, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1677560251009 in 281 msec 2023-02-28 04:57:34,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-28 04:57:34,352 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1677560251009, procId: 242 completed 2023-02-28 04:57:34,352 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1677560251009 table=testCorruptedSnapshot-1677560251009 type=FLUSH ttl=0 } 2023-02-28 04:57:34,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1677560254352 (current time:1677560254352). 2023-02-28 04:57:34,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-28 04:57:34,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] zookeeper.ReadOnlyZKClient(139): Connect 0x5ccd2066 to 127.0.0.1:54355 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-28 04:57:34,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4e21f0ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-28 04:57:34,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-28 04:57:34,363 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58356, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-28 04:57:34,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ccd2066 to 127.0.0.1:54355 2023-02-28 04:57:34,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:34,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-28 04:57:34,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1677560251009 table=testCorruptedSnapshot-1677560251009 type=FLUSH ttl=0 } 2023-02-28 04:57:34,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1677560251009, type=EXCLUSIVE 2023-02-28 04:57:34,366 DEBUG [PEWorker-3] locking.LockProcedure(309): LOCKED pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1677560251009, type=EXCLUSIVE 2023-02-28 04:57:34,367 INFO [PEWorker-3] procedure2.TimeoutExecutorThread(81): ADDED pid=255, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1677560251009, type=EXCLUSIVE; timeout=600000, timestamp=1677560854367 2023-02-28 04:57:34,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1677560251009 table=testCorruptedSnapshot-1677560251009 type=FLUSH ttl=0 } 2023-02-28 04:57:34,367 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1677560251009 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1677560251009 2023-02-28 04:57:34,367 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1677560251009 table=testCorruptedSnapshot-1677560251009 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-28 04:57:34,367 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-28 04:57:34,376 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1677560251009 table=testCorruptedSnapshot-1677560251009 type=DISABLED ttl=0 } 2023-02-28 04:57:34,378 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:34,378 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:34,378 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:34,378 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:34,378 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:34,378 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:34,378 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:34,378 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:34,378 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:34,378 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:34,378 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:34,378 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:34,378 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-28 04:57:34,378 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-28 04:57:34,380 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf/85b0830a8ec24e8e9d58e18a321b544e] hfiles 2023-02-28 04:57:34,380 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf/85b0830a8ec24e8e9d58e18a321b544e 2023-02-28 04:57:34,380 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf/648834311b524feb8715995350800fa8] hfiles 2023-02-28 04:57:34,380 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf/648834311b524feb8715995350800fa8 2023-02-28 04:57:34,380 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf/580c9ca6b54c4aa991c240159d3901be] hfiles 2023-02-28 04:57:34,380 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf/580c9ca6b54c4aa991c240159d3901be 2023-02-28 04:57:34,381 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf/a96fd165941e4d898d87f6cfa84457f8] hfiles 2023-02-28 04:57:34,381 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf/46c8c89265b5471ab4ddde49ca842fa6] hfiles 2023-02-28 04:57:34,381 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf/a96fd165941e4d898d87f6cfa84457f8 2023-02-28 04:57:34,381 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf/46c8c89265b5471ab4ddde49ca842fa6 2023-02-28 04:57:34,391 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf/d9d1454edff6491c896e22be29002893] hfiles 2023-02-28 04:57:34,391 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf/d9d1454edff6491c896e22be29002893 2023-02-28 04:57:34,393 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/a87ff679a2f3e71d9181a67b7542122c202302285080540a74f24d9296085cd67cf2022f, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/c4ca4238a0b923820dcc509a6f75849b20230228913dd8b6e2a34685aa1e0ea8bc127121, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/c81e728d9d4c2f636f067f89cc14862c202302283c19999ffdc44b829b28642a75d868ef, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/d41d8cd98f00b204e9800998ecf8427e2023022849be121c134747729f78fa30b34bf23d, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/e4da3b7fbbce2345d7772b0674a318d520230228cb27ffebb03648b289747a1cd18ac8e8, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302284ee665ca2c9e4e978c48fdb107b930b6] hfiles 2023-02-28 04:57:34,393 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/a87ff679a2f3e71d9181a67b7542122c202302285080540a74f24d9296085cd67cf2022f 2023-02-28 04:57:34,393 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/c4ca4238a0b923820dcc509a6f75849b20230228913dd8b6e2a34685aa1e0ea8bc127121 2023-02-28 04:57:34,393 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/c81e728d9d4c2f636f067f89cc14862c202302283c19999ffdc44b829b28642a75d868ef 2023-02-28 04:57:34,393 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/d41d8cd98f00b204e9800998ecf8427e2023022849be121c134747729f78fa30b34bf23d 2023-02-28 04:57:34,393 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/e4da3b7fbbce2345d7772b0674a318d520230228cb27ffebb03648b289747a1cd18ac8e8 2023-02-28 04:57:34,393 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302284ee665ca2c9e4e978c48fdb107b930b6 2023-02-28 04:57:34,405 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1677560251009 table=testCorruptedSnapshot-1677560251009 type=DISABLED ttl=0 } as finished. 2023-02-28 04:57:34,405 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1677560251009 2023-02-28 04:57:34,406 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/snaptb0-1677560251009 2023-02-28 04:57:34,424 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/snaptb0-1677560251009 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/snaptb0-1677560251009 2023-02-28 04:57:34,440 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1677560251009 of table testCorruptedSnapshot-1677560251009 completed 2023-02-28 04:57:34,440 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/snaptb0-1677560251009 2023-02-28 04:57:34,440 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.hbase-snapshot/.tmp/snaptb0-1677560251009 2023-02-28 04:57:34,440 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1677560251009 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1677560251009 at 1677560254367Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1677560251009 table=testCorruptedSnapshot-1677560251009 type=DISABLED ttl=0 } at 1677560254376 (+9 ms)Consolidate snapshot: snaptb0-1677560251009 at 1677560254405 (+29 ms)Loading Region manifests for snaptb0-1677560251009 at 1677560254405Writing data manifest for snaptb0-1677560251009 at 1677560254409 (+4 ms)Verifying snapshot: snaptb0-1677560251009 at 1677560254416 (+7 ms)Snapshot snaptb0-1677560251009 of table testCorruptedSnapshot-1677560251009 completed at 1677560254440 (+24 ms) 2023-02-28 04:57:34,441 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=255, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1677560251009, type=EXCLUSIVE 2023-02-28 04:57:34,442 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=255, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1677560251009, type=EXCLUSIVE in 76 msec 2023-02-28 04:57:34,618 DEBUG [Listener at localhost/36897] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-28 04:57:34,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1677560251009 table=testCorruptedSnapshot-1677560251009 type=FLUSH ttl=0 } is done 2023-02-28 04:57:34,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1677560251009 table=testCorruptedSnapshot-1677560251009 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-28 04:57:34,619 INFO [Listener at localhost/36897] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1677560251009 2023-02-28 04:57:34,619 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$10(2385): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1677560251009 2023-02-28 04:57:34,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=256, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1677560251009 2023-02-28 04:57:34,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-28 04:57:34,622 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1677560251009 2023-02-28 04:57:34,622 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560254622"}]},"ts":"1677560254622"} 2023-02-28 04:57:34,623 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1677560251009, state=ENABLING in hbase:meta 2023-02-28 04:57:34,626 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-28 04:57:34,626 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, ASSIGN}, {pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, ASSIGN}, {pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, ASSIGN}, {pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, ASSIGN}, {pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, ASSIGN}, {pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, ASSIGN}] 2023-02-28 04:57:34,627 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, ASSIGN 2023-02-28 04:57:34,627 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, ASSIGN 2023-02-28 04:57:34,627 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, ASSIGN 2023-02-28 04:57:34,627 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, ASSIGN 2023-02-28 04:57:34,628 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:34,628 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,36597,1677560224857; forceNewPlan=false, retain=true 2023-02-28 04:57:34,628 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:34,628 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:34,628 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, ASSIGN 2023-02-28 04:57:34,628 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:34,628 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,36597,1677560224857; forceNewPlan=false, retain=true 2023-02-28 04:57:34,628 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, ASSIGN 2023-02-28 04:57:34,628 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,38247,1677560224887; forceNewPlan=false, retain=true 2023-02-28 04:57:34,628 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46629,1677560224813; forceNewPlan=false, retain=true 2023-02-28 04:57:34,629 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:34,629 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,46629,1677560224813; forceNewPlan=false, retain=true 2023-02-28 04:57:34,629 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:34,629 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,38247,1677560224887; forceNewPlan=false, retain=true 2023-02-28 04:57:34,778 INFO [jenkins-hbase4:42505] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-28 04:57:34,778 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=35c83130614b8300868b867ad906a41e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:34,778 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=830ed62397c5222f09d35b7149e1b7db, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:34,778 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560254778"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560254778"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560254778"}]},"ts":"1677560254778"} 2023-02-28 04:57:34,778 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=6d657e535457f1d48270505f48f3ceda, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:34,778 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=9bc02e4052125a0d5af36b95e5b25a81, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:34,779 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560254778"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560254778"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560254778"}]},"ts":"1677560254778"} 2023-02-28 04:57:34,778 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=d67d3cb3ce05e60ff75f17313b64811d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:34,779 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560254778"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560254778"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560254778"}]},"ts":"1677560254778"} 2023-02-28 04:57:34,779 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560254778"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560254778"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560254778"}]},"ts":"1677560254778"} 2023-02-28 04:57:34,778 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560254778"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560254778"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560254778"}]},"ts":"1677560254778"} 2023-02-28 04:57:34,780 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=263, ppid=262, state=RUNNABLE; OpenRegionProcedure 35c83130614b8300868b867ad906a41e, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:34,781 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=264, ppid=261, state=RUNNABLE; OpenRegionProcedure 6d657e535457f1d48270505f48f3ceda, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:34,781 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=265, ppid=257, state=RUNNABLE; OpenRegionProcedure 9bc02e4052125a0d5af36b95e5b25a81, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:34,782 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=266, ppid=259, state=RUNNABLE; OpenRegionProcedure d67d3cb3ce05e60ff75f17313b64811d, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:34,783 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=267, ppid=258, state=RUNNABLE; OpenRegionProcedure 830ed62397c5222f09d35b7149e1b7db, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:34,783 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=640ee1a6dbf01ebb011a298d972247c6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:34,783 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560254783"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560254783"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560254783"}]},"ts":"1677560254783"} 2023-02-28 04:57:34,785 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=268, ppid=260, state=RUNNABLE; OpenRegionProcedure 640ee1a6dbf01ebb011a298d972247c6, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:34,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-28 04:57:34,933 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:34,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 35c83130614b8300868b867ad906a41e, NAME => 'testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.', STARTKEY => '5', ENDKEY => ''} 2023-02-28 04:57:34,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:34,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:34,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:34,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:34,934 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:34,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6d657e535457f1d48270505f48f3ceda, NAME => 'testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.', STARTKEY => '4', ENDKEY => '5'} 2023-02-28 04:57:34,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:34,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:34,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:34,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:34,934 INFO [StoreOpener-35c83130614b8300868b867ad906a41e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:34,935 INFO [StoreOpener-6d657e535457f1d48270505f48f3ceda-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:34,935 DEBUG [StoreOpener-35c83130614b8300868b867ad906a41e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf 2023-02-28 04:57:34,935 DEBUG [StoreOpener-35c83130614b8300868b867ad906a41e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf 2023-02-28 04:57:34,936 INFO [StoreOpener-35c83130614b8300868b867ad906a41e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 35c83130614b8300868b867ad906a41e columnFamilyName cf 2023-02-28 04:57:34,936 DEBUG [StoreOpener-6d657e535457f1d48270505f48f3ceda-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf 2023-02-28 04:57:34,936 DEBUG [StoreOpener-6d657e535457f1d48270505f48f3ceda-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf 2023-02-28 04:57:34,936 INFO [StoreOpener-6d657e535457f1d48270505f48f3ceda-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6d657e535457f1d48270505f48f3ceda columnFamilyName cf 2023-02-28 04:57:34,937 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:34,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 830ed62397c5222f09d35b7149e1b7db, NAME => 'testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.', STARTKEY => '1', ENDKEY => '2'} 2023-02-28 04:57:34,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:34,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:34,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:34,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:34,938 INFO [StoreOpener-830ed62397c5222f09d35b7149e1b7db-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:34,939 DEBUG [StoreOpener-830ed62397c5222f09d35b7149e1b7db-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf 2023-02-28 04:57:34,939 DEBUG [StoreOpener-830ed62397c5222f09d35b7149e1b7db-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf 2023-02-28 04:57:34,939 INFO [StoreOpener-830ed62397c5222f09d35b7149e1b7db-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 830ed62397c5222f09d35b7149e1b7db columnFamilyName cf 2023-02-28 04:57:34,940 DEBUG [StoreOpener-35c83130614b8300868b867ad906a41e-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf/85b0830a8ec24e8e9d58e18a321b544e 2023-02-28 04:57:34,940 DEBUG [StoreOpener-6d657e535457f1d48270505f48f3ceda-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf/648834311b524feb8715995350800fa8 2023-02-28 04:57:34,940 INFO [StoreOpener-6d657e535457f1d48270505f48f3ceda-1] regionserver.HStore(310): Store=6d657e535457f1d48270505f48f3ceda/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:34,940 INFO [StoreOpener-35c83130614b8300868b867ad906a41e-1] regionserver.HStore(310): Store=35c83130614b8300868b867ad906a41e/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:34,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e 2023-02-28 04:57:34,941 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:34,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:34,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e 2023-02-28 04:57:34,943 DEBUG [StoreOpener-830ed62397c5222f09d35b7149e1b7db-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf/46c8c89265b5471ab4ddde49ca842fa6 2023-02-28 04:57:34,943 INFO [StoreOpener-830ed62397c5222f09d35b7149e1b7db-1] regionserver.HStore(310): Store=830ed62397c5222f09d35b7149e1b7db/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:34,943 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:34,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:34,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:34,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:34,945 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 35c83130614b8300868b867ad906a41e; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58732322, jitterRate=-0.12482020258903503}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:34,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 35c83130614b8300868b867ad906a41e: 2023-02-28 04:57:34,945 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 6d657e535457f1d48270505f48f3ceda; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62470294, jitterRate=-0.06912007927894592}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:34,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 6d657e535457f1d48270505f48f3ceda: 2023-02-28 04:57:34,946 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e., pid=263, masterSystemTime=1677560254931 2023-02-28 04:57:34,946 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda., pid=264, masterSystemTime=1677560254932 2023-02-28 04:57:34,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:34,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:34,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:34,948 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:34,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d67d3cb3ce05e60ff75f17313b64811d, NAME => 'testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.', STARTKEY => '2', ENDKEY => '3'} 2023-02-28 04:57:34,948 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 830ed62397c5222f09d35b7149e1b7db; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71552158, jitterRate=0.06621024012565613}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:34,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 830ed62397c5222f09d35b7149e1b7db: 2023-02-28 04:57:34,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:34,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:34,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:34,948 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:34,948 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=35c83130614b8300868b867ad906a41e, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:34,948 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db., pid=267, masterSystemTime=1677560254935 2023-02-28 04:57:34,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:34,949 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560254948"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560254948"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560254948"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560254948"}]},"ts":"1677560254948"} 2023-02-28 04:57:34,949 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:34,949 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:34,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 9bc02e4052125a0d5af36b95e5b25a81, NAME => 'testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.', STARTKEY => '', ENDKEY => '1'} 2023-02-28 04:57:34,949 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=6d657e535457f1d48270505f48f3ceda, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:34,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:34,949 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560254949"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560254949"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560254949"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560254949"}]},"ts":"1677560254949"} 2023-02-28 04:57:34,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:34,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:34,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:34,950 INFO [StoreOpener-d67d3cb3ce05e60ff75f17313b64811d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:34,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:34,950 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:34,950 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:34,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 640ee1a6dbf01ebb011a298d972247c6, NAME => 'testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.', STARTKEY => '3', ENDKEY => '4'} 2023-02-28 04:57:34,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1677560251009 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:34,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-28 04:57:34,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:34,950 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:34,951 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=830ed62397c5222f09d35b7149e1b7db, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:34,951 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560254950"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560254950"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560254950"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560254950"}]},"ts":"1677560254950"} 2023-02-28 04:57:34,952 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=263, resume processing ppid=262 2023-02-28 04:57:34,952 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=263, ppid=262, state=SUCCESS; OpenRegionProcedure 35c83130614b8300868b867ad906a41e, server=jenkins-hbase4.apache.org,46629,1677560224813 in 170 msec 2023-02-28 04:57:34,953 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=264, resume processing ppid=261 2023-02-28 04:57:34,953 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=262, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, ASSIGN in 326 msec 2023-02-28 04:57:34,953 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=264, ppid=261, state=SUCCESS; OpenRegionProcedure 6d657e535457f1d48270505f48f3ceda, server=jenkins-hbase4.apache.org,38247,1677560224887 in 170 msec 2023-02-28 04:57:34,953 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=267, resume processing ppid=258 2023-02-28 04:57:34,953 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=267, ppid=258, state=SUCCESS; OpenRegionProcedure 830ed62397c5222f09d35b7149e1b7db, server=jenkins-hbase4.apache.org,36597,1677560224857 in 169 msec 2023-02-28 04:57:34,953 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=261, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, ASSIGN in 327 msec 2023-02-28 04:57:34,954 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=258, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, ASSIGN in 327 msec 2023-02-28 04:57:34,956 INFO [StoreOpener-640ee1a6dbf01ebb011a298d972247c6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:34,956 DEBUG [StoreOpener-d67d3cb3ce05e60ff75f17313b64811d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf 2023-02-28 04:57:34,956 DEBUG [StoreOpener-d67d3cb3ce05e60ff75f17313b64811d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf 2023-02-28 04:57:34,957 INFO [StoreOpener-9bc02e4052125a0d5af36b95e5b25a81-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:34,957 INFO [StoreOpener-d67d3cb3ce05e60ff75f17313b64811d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d67d3cb3ce05e60ff75f17313b64811d columnFamilyName cf 2023-02-28 04:57:34,957 DEBUG [StoreOpener-640ee1a6dbf01ebb011a298d972247c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf 2023-02-28 04:57:34,957 DEBUG [StoreOpener-640ee1a6dbf01ebb011a298d972247c6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf 2023-02-28 04:57:34,957 DEBUG [StoreOpener-9bc02e4052125a0d5af36b95e5b25a81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf 2023-02-28 04:57:34,957 DEBUG [StoreOpener-9bc02e4052125a0d5af36b95e5b25a81-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf 2023-02-28 04:57:34,958 INFO [StoreOpener-640ee1a6dbf01ebb011a298d972247c6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 640ee1a6dbf01ebb011a298d972247c6 columnFamilyName cf 2023-02-28 04:57:34,958 INFO [StoreOpener-9bc02e4052125a0d5af36b95e5b25a81-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9bc02e4052125a0d5af36b95e5b25a81 columnFamilyName cf 2023-02-28 04:57:34,966 DEBUG [StoreOpener-d67d3cb3ce05e60ff75f17313b64811d-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf/a96fd165941e4d898d87f6cfa84457f8 2023-02-28 04:57:34,966 DEBUG [StoreOpener-640ee1a6dbf01ebb011a298d972247c6-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf/d9d1454edff6491c896e22be29002893 2023-02-28 04:57:34,966 INFO [StoreOpener-d67d3cb3ce05e60ff75f17313b64811d-1] regionserver.HStore(310): Store=d67d3cb3ce05e60ff75f17313b64811d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:34,966 INFO [StoreOpener-640ee1a6dbf01ebb011a298d972247c6-1] regionserver.HStore(310): Store=640ee1a6dbf01ebb011a298d972247c6/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:34,966 DEBUG [StoreOpener-9bc02e4052125a0d5af36b95e5b25a81-1] regionserver.HStore(539): loaded hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf/580c9ca6b54c4aa991c240159d3901be 2023-02-28 04:57:34,966 INFO [StoreOpener-9bc02e4052125a0d5af36b95e5b25a81-1] regionserver.HStore(310): Store=9bc02e4052125a0d5af36b95e5b25a81/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-28 04:57:34,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:34,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:34,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:34,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:34,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:34,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:34,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:34,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:34,970 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:34,971 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 640ee1a6dbf01ebb011a298d972247c6; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63656235, jitterRate=-0.05144818127155304}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:34,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 640ee1a6dbf01ebb011a298d972247c6: 2023-02-28 04:57:34,971 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d67d3cb3ce05e60ff75f17313b64811d; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58996022, jitterRate=-0.12089076638221741}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:34,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d67d3cb3ce05e60ff75f17313b64811d: 2023-02-28 04:57:34,971 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 9bc02e4052125a0d5af36b95e5b25a81; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74859314, jitterRate=0.11549070477485657}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-28 04:57:34,971 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 9bc02e4052125a0d5af36b95e5b25a81: 2023-02-28 04:57:34,971 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81., pid=265, masterSystemTime=1677560254932 2023-02-28 04:57:34,972 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d., pid=266, masterSystemTime=1677560254931 2023-02-28 04:57:34,972 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6., pid=268, masterSystemTime=1677560254935 2023-02-28 04:57:34,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:34,973 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:34,973 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=9bc02e4052125a0d5af36b95e5b25a81, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:34,973 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560254973"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560254973"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560254973"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560254973"}]},"ts":"1677560254973"} 2023-02-28 04:57:34,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:34,973 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:34,974 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=d67d3cb3ce05e60ff75f17313b64811d, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:34,974 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560254974"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560254974"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560254974"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560254974"}]},"ts":"1677560254974"} 2023-02-28 04:57:34,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:34,974 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:34,974 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=640ee1a6dbf01ebb011a298d972247c6, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:34,974 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560254974"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1677560254974"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1677560254974"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1677560254974"}]},"ts":"1677560254974"} 2023-02-28 04:57:34,976 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=265, resume processing ppid=257 2023-02-28 04:57:34,976 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=265, ppid=257, state=SUCCESS; OpenRegionProcedure 9bc02e4052125a0d5af36b95e5b25a81, server=jenkins-hbase4.apache.org,38247,1677560224887 in 193 msec 2023-02-28 04:57:34,977 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=266, resume processing ppid=259 2023-02-28 04:57:34,977 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=266, ppid=259, state=SUCCESS; OpenRegionProcedure d67d3cb3ce05e60ff75f17313b64811d, server=jenkins-hbase4.apache.org,46629,1677560224813 in 193 msec 2023-02-28 04:57:34,977 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=257, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, ASSIGN in 350 msec 2023-02-28 04:57:34,978 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=268, resume processing ppid=260 2023-02-28 04:57:34,978 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=268, ppid=260, state=SUCCESS; OpenRegionProcedure 640ee1a6dbf01ebb011a298d972247c6, server=jenkins-hbase4.apache.org,36597,1677560224857 in 191 msec 2023-02-28 04:57:34,978 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=259, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, ASSIGN in 351 msec 2023-02-28 04:57:34,979 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=260, resume processing ppid=256 2023-02-28 04:57:34,979 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=260, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, ASSIGN in 352 msec 2023-02-28 04:57:34,980 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560254979"}]},"ts":"1677560254979"} 2023-02-28 04:57:34,980 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1677560251009, state=ENABLED in hbase:meta 2023-02-28 04:57:34,983 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1677560251009' was successfully enabled. 2023-02-28 04:57:34,984 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=256, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1677560251009 in 364 msec 2023-02-28 04:57:35,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-28 04:57:35,374 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1677560251009, procId: 256 completed 2023-02-28 04:57:35,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:35,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:35,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38247] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:35,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36597] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:35,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:35,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46629] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. with WAL disabled. Data may be lost in the event of a crash. 2023-02-28 04:57:35,388 DEBUG [Listener at localhost/36897] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1677560251009 2023-02-28 04:57:35,388 DEBUG [Listener at localhost/36897] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:35,388 INFO [Listener at localhost/36897] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-28 04:57:35,460 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-28 04:57:35,542 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-28 04:57:35,629 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x594888d9 to 127.0.0.1:54355 2023-02-28 04:57:35,629 DEBUG [Finalizer] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:35,658 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-28 04:57:35,722 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-28 04:57:35,778 DEBUG [Listener at localhost/36897] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-28 04:57:36,611 ERROR [VerifySnapshot-pool-0] snapshot.SnapshotReferenceUtil(301): Can't find hfile: 85b0830a8ec24e8e9d58e18a321b544e in the real (hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf/85b0830a8ec24e8e9d58e18a321b544e) or archive (hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf/85b0830a8ec24e8e9d58e18a321b544e) directory for the primary table. 2023-02-28 04:57:36,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 85b0830a8ec24e8e9d58e18a321b544e in the real (hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf/85b0830a8ec24e8e9d58e18a321b544e) or archive (hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf/85b0830a8ec24e8e9d58e18a321b544e) directory for the primary table. at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:225) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:187) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.verifySnapshot(SnapshotReferenceUtil.java:164) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:827) at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2506) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132) at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2502) at org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1469) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-28 04:57:36,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] ipc.CallRunner(144): callId: 358 service: MasterService methodName: RestoreSnapshot size: 115 connection: 172.31.14.131:33426 deadline: 1677560316607, exception=org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: 85b0830a8ec24e8e9d58e18a321b544e in the real (hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf/85b0830a8ec24e8e9d58e18a321b544e) or archive (hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf/85b0830a8ec24e8e9d58e18a321b544e) directory for the primary table. 2023-02-28 04:57:36,630 INFO [Listener at localhost/36897] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1677560251009 2023-02-28 04:57:36,630 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1677560251009 2023-02-28 04:57:36,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=270, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1677560251009 2023-02-28 04:57:36,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-28 04:57:36,634 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560256633"}]},"ts":"1677560256633"} 2023-02-28 04:57:36,635 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1677560251009, state=DISABLING in hbase:meta 2023-02-28 04:57:36,636 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1677560251009 to state=DISABLING 2023-02-28 04:57:36,637 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, UNASSIGN}, {pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, UNASSIGN}, {pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, UNASSIGN}, {pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, UNASSIGN}, {pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, UNASSIGN}, {pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, UNASSIGN}] 2023-02-28 04:57:36,638 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, UNASSIGN 2023-02-28 04:57:36,638 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, UNASSIGN 2023-02-28 04:57:36,638 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, UNASSIGN 2023-02-28 04:57:36,638 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, UNASSIGN 2023-02-28 04:57:36,638 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, UNASSIGN 2023-02-28 04:57:36,639 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=640ee1a6dbf01ebb011a298d972247c6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:36,639 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560256639"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560256639"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560256639"}]},"ts":"1677560256639"} 2023-02-28 04:57:36,639 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=830ed62397c5222f09d35b7149e1b7db, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:36,639 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=35c83130614b8300868b867ad906a41e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:36,639 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560256639"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560256639"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560256639"}]},"ts":"1677560256639"} 2023-02-28 04:57:36,639 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=d67d3cb3ce05e60ff75f17313b64811d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:36,639 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=9bc02e4052125a0d5af36b95e5b25a81, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:36,639 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560256639"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560256639"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560256639"}]},"ts":"1677560256639"} 2023-02-28 04:57:36,639 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560256639"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560256639"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560256639"}]},"ts":"1677560256639"} 2023-02-28 04:57:36,639 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560256639"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560256639"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560256639"}]},"ts":"1677560256639"} 2023-02-28 04:57:36,641 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=277, ppid=274, state=RUNNABLE; CloseRegionProcedure 640ee1a6dbf01ebb011a298d972247c6, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:36,641 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=278, ppid=272, state=RUNNABLE; CloseRegionProcedure 830ed62397c5222f09d35b7149e1b7db, server=jenkins-hbase4.apache.org,36597,1677560224857}] 2023-02-28 04:57:36,642 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=279, ppid=271, state=RUNNABLE; CloseRegionProcedure 9bc02e4052125a0d5af36b95e5b25a81, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:36,643 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=280, ppid=276, state=RUNNABLE; CloseRegionProcedure 35c83130614b8300868b867ad906a41e, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:36,644 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=281, ppid=273, state=RUNNABLE; CloseRegionProcedure d67d3cb3ce05e60ff75f17313b64811d, server=jenkins-hbase4.apache.org,46629,1677560224813}] 2023-02-28 04:57:36,644 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, UNASSIGN 2023-02-28 04:57:36,645 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=6d657e535457f1d48270505f48f3ceda, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:36,645 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560256645"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1677560256645"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1677560256645"}]},"ts":"1677560256645"} 2023-02-28 04:57:36,646 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=282, ppid=275, state=RUNNABLE; CloseRegionProcedure 6d657e535457f1d48270505f48f3ceda, server=jenkins-hbase4.apache.org,38247,1677560224887}] 2023-02-28 04:57:36,793 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:36,793 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 830ed62397c5222f09d35b7149e1b7db, disabling compactions & flushes 2023-02-28 04:57:36,793 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:36,793 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:36,793 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. after waiting 0 ms 2023-02-28 04:57:36,793 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:36,793 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 830ed62397c5222f09d35b7149e1b7db 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-28 04:57:36,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:36,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:36,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 6d657e535457f1d48270505f48f3ceda, disabling compactions & flushes 2023-02-28 04:57:36,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d67d3cb3ce05e60ff75f17313b64811d, disabling compactions & flushes 2023-02-28 04:57:36,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:36,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:36,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:36,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. after waiting 0 ms 2023-02-28 04:57:36,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:36,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 6d657e535457f1d48270505f48f3ceda 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-28 04:57:36,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:36,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. after waiting 0 ms 2023-02-28 04:57:36,796 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:36,796 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing d67d3cb3ce05e60ff75f17313b64811d 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-28 04:57:36,813 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302289fa99a8f49b54c27a09d4c0f517e206f to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/a87ff679a2f3e71d9181a67b7542122c202302289fa99a8f49b54c27a09d4c0f517e206f 2023-02-28 04:57:36,813 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b202302288644d2a32daa424b80ceb63c3713931a to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/c4ca4238a0b923820dcc509a6f75849b202302288644d2a32daa424b80ceb63c3713931a 2023-02-28 04:57:36,813 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230228d0d9ed5de5034f14ad9a5b75e7060f9a to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/c81e728d9d4c2f636f067f89cc14862c20230228d0d9ed5de5034f14ad9a5b75e7060f9a 2023-02-28 04:57:36,822 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/.tmp/cf/1a7fa21256614a57aa95dd18df04b605 2023-02-28 04:57:36,824 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/.tmp/cf/1c0a7df173bc42e9a69aafa9aca671d6 2023-02-28 04:57:36,824 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/.tmp/cf/c0a78ea013e94005b75c14d7a2b5b8d8 2023-02-28 04:57:36,826 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/.tmp/cf/1a7fa21256614a57aa95dd18df04b605 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf/1a7fa21256614a57aa95dd18df04b605 2023-02-28 04:57:36,827 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/.tmp/cf/1c0a7df173bc42e9a69aafa9aca671d6 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf/1c0a7df173bc42e9a69aafa9aca671d6 2023-02-28 04:57:36,827 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/.tmp/cf/c0a78ea013e94005b75c14d7a2b5b8d8 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf/c0a78ea013e94005b75c14d7a2b5b8d8 2023-02-28 04:57:36,830 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf/1c0a7df173bc42e9a69aafa9aca671d6, entries=27, sequenceid=15, filesize=9.6 K 2023-02-28 04:57:36,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for d67d3cb3ce05e60ff75f17313b64811d in 34ms, sequenceid=15, compaction requested=false 2023-02-28 04:57:36,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf/c0a78ea013e94005b75c14d7a2b5b8d8, entries=26, sequenceid=15, filesize=9.4 K 2023-02-28 04:57:36,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf/1a7fa21256614a57aa95dd18df04b605, entries=31, sequenceid=15, filesize=10.2 K 2023-02-28 04:57:36,834 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 6d657e535457f1d48270505f48f3ceda in 38ms, sequenceid=15, compaction requested=false 2023-02-28 04:57:36,834 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for 830ed62397c5222f09d35b7149e1b7db in 38ms, sequenceid=15, compaction requested=false 2023-02-28 04:57:36,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-28 04:57:36,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d. 2023-02-28 04:57:36,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d67d3cb3ce05e60ff75f17313b64811d: 2023-02-28 04:57:36,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-28 04:57:36,837 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda. 2023-02-28 04:57:36,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 6d657e535457f1d48270505f48f3ceda: 2023-02-28 04:57:36,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:36,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:36,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 35c83130614b8300868b867ad906a41e, disabling compactions & flushes 2023-02-28 04:57:36,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:36,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:36,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-28 04:57:36,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. after waiting 0 ms 2023-02-28 04:57:36,838 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=d67d3cb3ce05e60ff75f17313b64811d, regionState=CLOSED 2023-02-28 04:57:36,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:36,838 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560256838"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560256838"}]},"ts":"1677560256838"} 2023-02-28 04:57:36,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:36,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 35c83130614b8300868b867ad906a41e 1/1 column families, dataSize=22.77 KB heapSize=49.19 KB 2023-02-28 04:57:36,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db. 2023-02-28 04:57:36,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:36,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 830ed62397c5222f09d35b7149e1b7db: 2023-02-28 04:57:36,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 9bc02e4052125a0d5af36b95e5b25a81, disabling compactions & flushes 2023-02-28 04:57:36,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:36,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:36,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. after waiting 0 ms 2023-02-28 04:57:36,839 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:36,839 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 9bc02e4052125a0d5af36b95e5b25a81 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-28 04:57:36,840 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=6d657e535457f1d48270505f48f3ceda, regionState=CLOSED 2023-02-28 04:57:36,840 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560256840"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560256840"}]},"ts":"1677560256840"} 2023-02-28 04:57:36,842 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:36,842 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:36,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 640ee1a6dbf01ebb011a298d972247c6, disabling compactions & flushes 2023-02-28 04:57:36,843 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:36,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:36,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. after waiting 0 ms 2023-02-28 04:57:36,843 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:36,843 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 640ee1a6dbf01ebb011a298d972247c6 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-28 04:57:36,843 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=830ed62397c5222f09d35b7149e1b7db, regionState=CLOSED 2023-02-28 04:57:36,843 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560256843"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560256843"}]},"ts":"1677560256843"} 2023-02-28 04:57:36,846 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=281, resume processing ppid=273 2023-02-28 04:57:36,846 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=281, ppid=273, state=SUCCESS; CloseRegionProcedure d67d3cb3ce05e60ff75f17313b64811d, server=jenkins-hbase4.apache.org,46629,1677560224813 in 198 msec 2023-02-28 04:57:36,846 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=282, resume processing ppid=275 2023-02-28 04:57:36,846 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=282, ppid=275, state=SUCCESS; CloseRegionProcedure 6d657e535457f1d48270505f48f3ceda, server=jenkins-hbase4.apache.org,38247,1677560224887 in 198 msec 2023-02-28 04:57:36,847 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=278, resume processing ppid=272 2023-02-28 04:57:36,847 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=273, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=d67d3cb3ce05e60ff75f17313b64811d, UNASSIGN in 209 msec 2023-02-28 04:57:36,847 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=278, ppid=272, state=SUCCESS; CloseRegionProcedure 830ed62397c5222f09d35b7149e1b7db, server=jenkins-hbase4.apache.org,36597,1677560224857 in 204 msec 2023-02-28 04:57:36,847 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=275, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=6d657e535457f1d48270505f48f3ceda, UNASSIGN in 209 msec 2023-02-28 04:57:36,848 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=272, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=830ed62397c5222f09d35b7149e1b7db, UNASSIGN in 210 msec 2023-02-28 04:57:36,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023022856bb671c7aea4cab99c505386d5467fd to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/d41d8cd98f00b204e9800998ecf8427e2023022856bb671c7aea4cab99c505386d5467fd 2023-02-28 04:57:36,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf3202302284497ce8475834b678b7652ac221e6ec4 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302284497ce8475834b678b7652ac221e6ec4 2023-02-28 04:57:36,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d520230228acfbc467b0b64bcd81eb1c234125bf20 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/e4da3b7fbbce2345d7772b0674a318d520230228acfbc467b0b64bcd81eb1c234125bf20 2023-02-28 04:57:36,878 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/.tmp/cf/e39dacd98c0c4e7c8efc64e5974f6033 2023-02-28 04:57:36,881 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/.tmp/cf/e39dacd98c0c4e7c8efc64e5974f6033 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf/e39dacd98c0c4e7c8efc64e5974f6033 2023-02-28 04:57:36,884 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf/e39dacd98c0c4e7c8efc64e5974f6033, entries=32, sequenceid=15, filesize=10.4 K 2023-02-28 04:57:36,884 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for 9bc02e4052125a0d5af36b95e5b25a81 in 45ms, sequenceid=15, compaction requested=false 2023-02-28 04:57:36,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-28 04:57:36,888 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/.tmp/cf/1080e44de4e04262b8127b43b56a2656 2023-02-28 04:57:36,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/.tmp/cf/1080e44de4e04262b8127b43b56a2656 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf/1080e44de4e04262b8127b43b56a2656 2023-02-28 04:57:36,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf/1080e44de4e04262b8127b43b56a2656, entries=36, sequenceid=15, filesize=11.1 K 2023-02-28 04:57:36,895 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 640ee1a6dbf01ebb011a298d972247c6 in 52ms, sequenceid=15, compaction requested=false 2023-02-28 04:57:36,895 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-28 04:57:36,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81. 2023-02-28 04:57:36,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 9bc02e4052125a0d5af36b95e5b25a81: 2023-02-28 04:57:36,898 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:36,898 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=9bc02e4052125a0d5af36b95e5b25a81, regionState=CLOSED 2023-02-28 04:57:36,898 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560256898"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560256898"}]},"ts":"1677560256898"} 2023-02-28 04:57:36,901 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=279, resume processing ppid=271 2023-02-28 04:57:36,901 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=279, ppid=271, state=SUCCESS; CloseRegionProcedure 9bc02e4052125a0d5af36b95e5b25a81, server=jenkins-hbase4.apache.org,38247,1677560224887 in 258 msec 2023-02-28 04:57:36,902 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=271, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=9bc02e4052125a0d5af36b95e5b25a81, UNASSIGN in 264 msec 2023-02-28 04:57:36,904 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-28 04:57:36,905 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6. 2023-02-28 04:57:36,905 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 640ee1a6dbf01ebb011a298d972247c6: 2023-02-28 04:57:36,906 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:36,906 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=640ee1a6dbf01ebb011a298d972247c6, regionState=CLOSED 2023-02-28 04:57:36,906 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1677560256906"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560256906"}]},"ts":"1677560256906"} 2023-02-28 04:57:36,908 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=277, resume processing ppid=274 2023-02-28 04:57:36,909 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=277, ppid=274, state=SUCCESS; CloseRegionProcedure 640ee1a6dbf01ebb011a298d972247c6, server=jenkins-hbase4.apache.org,36597,1677560224857 in 266 msec 2023-02-28 04:57:36,909 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=274, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=640ee1a6dbf01ebb011a298d972247c6, UNASSIGN in 272 msec 2023-02-28 04:57:37,301 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=22.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/.tmp/cf/cfe45e0a4e014f59930075dbe6389e08 2023-02-28 04:57:37,305 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/.tmp/cf/cfe45e0a4e014f59930075dbe6389e08 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf/cfe45e0a4e014f59930075dbe6389e08 2023-02-28 04:57:37,307 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf/cfe45e0a4e014f59930075dbe6389e08, entries=348, sequenceid=15, filesize=64.9 K 2023-02-28 04:57:37,308 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.77 KB/23312, heapSize ~49.17 KB/50352, currentSize=0 B/0 for 35c83130614b8300868b867ad906a41e in 470ms, sequenceid=15, compaction requested=false 2023-02-28 04:57:37,311 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-28 04:57:37,311 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e. 2023-02-28 04:57:37,311 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 35c83130614b8300868b867ad906a41e: 2023-02-28 04:57:37,312 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 35c83130614b8300868b867ad906a41e 2023-02-28 04:57:37,312 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=35c83130614b8300868b867ad906a41e, regionState=CLOSED 2023-02-28 04:57:37,312 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1677560257312"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1677560257312"}]},"ts":"1677560257312"} 2023-02-28 04:57:37,314 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=280, resume processing ppid=276 2023-02-28 04:57:37,314 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=280, ppid=276, state=SUCCESS; CloseRegionProcedure 35c83130614b8300868b867ad906a41e, server=jenkins-hbase4.apache.org,46629,1677560224813 in 670 msec 2023-02-28 04:57:37,315 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=276, resume processing ppid=270 2023-02-28 04:57:37,315 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=276, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1677560251009, region=35c83130614b8300868b867ad906a41e, UNASSIGN in 677 msec 2023-02-28 04:57:37,316 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1677560257316"}]},"ts":"1677560257316"} 2023-02-28 04:57:37,317 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1677560251009, state=DISABLED in hbase:meta 2023-02-28 04:57:37,318 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1677560251009 to state=DISABLED 2023-02-28 04:57:37,319 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=270, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1677560251009 in 689 msec 2023-02-28 04:57:37,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-28 04:57:37,386 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1677560251009, procId: 270 completed 2023-02-28 04:57:37,387 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testCorruptedSnapshot-1677560251009 2023-02-28 04:57:37,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] procedure2.ProcedureExecutor(1029): Stored pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCorruptedSnapshot-1677560251009 2023-02-28 04:57:37,389 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1677560251009 2023-02-28 04:57:37,390 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=283, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1677560251009 2023-02-28 04:57:37,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-28 04:57:37,396 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:37,396 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:37,396 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e 2023-02-28 04:57:37,396 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:37,396 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:37,396 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:37,398 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/recovered.edits] 2023-02-28 04:57:37,399 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/recovered.edits] 2023-02-28 04:57:37,399 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/recovered.edits] 2023-02-28 04:57:37,399 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/recovered.edits] 2023-02-28 04:57:37,400 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/recovered.edits] 2023-02-28 04:57:37,400 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf, FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/recovered.edits] 2023-02-28 04:57:37,404 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf/580c9ca6b54c4aa991c240159d3901be to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf/580c9ca6b54c4aa991c240159d3901be 2023-02-28 04:57:37,405 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf/1a7fa21256614a57aa95dd18df04b605 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf/1a7fa21256614a57aa95dd18df04b605 2023-02-28 04:57:37,405 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf/1c0a7df173bc42e9a69aafa9aca671d6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf/1c0a7df173bc42e9a69aafa9aca671d6 2023-02-28 04:57:37,406 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf/1080e44de4e04262b8127b43b56a2656 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf/1080e44de4e04262b8127b43b56a2656 2023-02-28 04:57:37,406 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf/e39dacd98c0c4e7c8efc64e5974f6033 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/cf/e39dacd98c0c4e7c8efc64e5974f6033 2023-02-28 04:57:37,406 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf/46c8c89265b5471ab4ddde49ca842fa6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf/46c8c89265b5471ab4ddde49ca842fa6 2023-02-28 04:57:37,406 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf/648834311b524feb8715995350800fa8 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/cf/648834311b524feb8715995350800fa8 2023-02-28 04:57:37,407 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf/cfe45e0a4e014f59930075dbe6389e08 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/cf/cfe45e0a4e014f59930075dbe6389e08 2023-02-28 04:57:37,407 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf/a96fd165941e4d898d87f6cfa84457f8 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/cf/a96fd165941e4d898d87f6cfa84457f8 2023-02-28 04:57:37,407 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf/d9d1454edff6491c896e22be29002893 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/cf/d9d1454edff6491c896e22be29002893 2023-02-28 04:57:37,408 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf/c0a78ea013e94005b75c14d7a2b5b8d8 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/cf/c0a78ea013e94005b75c14d7a2b5b8d8 2023-02-28 04:57:37,410 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/recovered.edits/18.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda/recovered.edits/18.seqid 2023-02-28 04:57:37,410 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/recovered.edits/18.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d/recovered.edits/18.seqid 2023-02-28 04:57:37,411 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/recovered.edits/18.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e/recovered.edits/18.seqid 2023-02-28 04:57:37,411 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/recovered.edits/18.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81/recovered.edits/18.seqid 2023-02-28 04:57:37,411 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/6d657e535457f1d48270505f48f3ceda 2023-02-28 04:57:37,411 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/recovered.edits/18.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6/recovered.edits/18.seqid 2023-02-28 04:57:37,411 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/d67d3cb3ce05e60ff75f17313b64811d 2023-02-28 04:57:37,411 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/35c83130614b8300868b867ad906a41e 2023-02-28 04:57:37,412 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/9bc02e4052125a0d5af36b95e5b25a81 2023-02-28 04:57:37,412 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/640ee1a6dbf01ebb011a298d972247c6 2023-02-28 04:57:37,412 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/recovered.edits/18.seqid to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db/recovered.edits/18.seqid 2023-02-28 04:57:37,413 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/.tmp/data/default/testCorruptedSnapshot-1677560251009/830ed62397c5222f09d35b7149e1b7db 2023-02-28 04:57:37,413 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1677560251009 regions 2023-02-28 04:57:37,413 DEBUG [PEWorker-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489 2023-02-28 04:57:37,413 DEBUG [PEWorker-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf] 2023-02-28 04:57:37,416 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/a87ff679a2f3e71d9181a67b7542122c202302285080540a74f24d9296085cd67cf2022f to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/a87ff679a2f3e71d9181a67b7542122c202302285080540a74f24d9296085cd67cf2022f 2023-02-28 04:57:37,417 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/a87ff679a2f3e71d9181a67b7542122c202302289fa99a8f49b54c27a09d4c0f517e206f to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/a87ff679a2f3e71d9181a67b7542122c202302289fa99a8f49b54c27a09d4c0f517e206f 2023-02-28 04:57:37,418 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/c4ca4238a0b923820dcc509a6f75849b202302288644d2a32daa424b80ceb63c3713931a to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/c4ca4238a0b923820dcc509a6f75849b202302288644d2a32daa424b80ceb63c3713931a 2023-02-28 04:57:37,419 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/c4ca4238a0b923820dcc509a6f75849b20230228913dd8b6e2a34685aa1e0ea8bc127121 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/c4ca4238a0b923820dcc509a6f75849b20230228913dd8b6e2a34685aa1e0ea8bc127121 2023-02-28 04:57:37,420 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/c81e728d9d4c2f636f067f89cc14862c202302283c19999ffdc44b829b28642a75d868ef to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/c81e728d9d4c2f636f067f89cc14862c202302283c19999ffdc44b829b28642a75d868ef 2023-02-28 04:57:37,421 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/c81e728d9d4c2f636f067f89cc14862c20230228d0d9ed5de5034f14ad9a5b75e7060f9a to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/c81e728d9d4c2f636f067f89cc14862c20230228d0d9ed5de5034f14ad9a5b75e7060f9a 2023-02-28 04:57:37,422 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/d41d8cd98f00b204e9800998ecf8427e2023022849be121c134747729f78fa30b34bf23d to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/d41d8cd98f00b204e9800998ecf8427e2023022849be121c134747729f78fa30b34bf23d 2023-02-28 04:57:37,422 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/d41d8cd98f00b204e9800998ecf8427e2023022856bb671c7aea4cab99c505386d5467fd to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/d41d8cd98f00b204e9800998ecf8427e2023022856bb671c7aea4cab99c505386d5467fd 2023-02-28 04:57:37,423 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/e4da3b7fbbce2345d7772b0674a318d520230228acfbc467b0b64bcd81eb1c234125bf20 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/e4da3b7fbbce2345d7772b0674a318d520230228acfbc467b0b64bcd81eb1c234125bf20 2023-02-28 04:57:37,424 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/e4da3b7fbbce2345d7772b0674a318d520230228cb27ffebb03648b289747a1cd18ac8e8 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/e4da3b7fbbce2345d7772b0674a318d520230228cb27ffebb03648b289747a1cd18ac8e8 2023-02-28 04:57:37,425 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302284497ce8475834b678b7652ac221e6ec4 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302284497ce8475834b678b7652ac221e6ec4 2023-02-28 04:57:37,426 DEBUG [PEWorker-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302284ee665ca2c9e4e978c48fdb107b930b6 to hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/archive/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302284ee665ca2c9e4e978c48fdb107b930b6 2023-02-28 04:57:37,426 DEBUG [PEWorker-2] backup.HFileArchiver(596): Deleted hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/mobdir/data/default/testCorruptedSnapshot-1677560251009/e58d211bd8ef5dd3bfb472568a598489 2023-02-28 04:57:37,428 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=283, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1677560251009 2023-02-28 04:57:37,430 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testCorruptedSnapshot-1677560251009 from hbase:meta 2023-02-28 04:57:37,432 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'testCorruptedSnapshot-1677560251009' descriptor. 2023-02-28 04:57:37,433 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=283, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1677560251009 2023-02-28 04:57:37,433 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'testCorruptedSnapshot-1677560251009' from region states. 2023-02-28 04:57:37,433 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560257433"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:37,433 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560257433"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:37,433 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560257433"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:37,433 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560257433"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:37,433 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560257433"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:37,433 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1677560257433"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:37,435 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-28 04:57:37,435 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 9bc02e4052125a0d5af36b95e5b25a81, NAME => 'testCorruptedSnapshot-1677560251009,,1677560251011.9bc02e4052125a0d5af36b95e5b25a81.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => 830ed62397c5222f09d35b7149e1b7db, NAME => 'testCorruptedSnapshot-1677560251009,1,1677560251011.830ed62397c5222f09d35b7149e1b7db.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => d67d3cb3ce05e60ff75f17313b64811d, NAME => 'testCorruptedSnapshot-1677560251009,2,1677560251011.d67d3cb3ce05e60ff75f17313b64811d.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 640ee1a6dbf01ebb011a298d972247c6, NAME => 'testCorruptedSnapshot-1677560251009,3,1677560251011.640ee1a6dbf01ebb011a298d972247c6.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 6d657e535457f1d48270505f48f3ceda, NAME => 'testCorruptedSnapshot-1677560251009,4,1677560251011.6d657e535457f1d48270505f48f3ceda.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 35c83130614b8300868b867ad906a41e, NAME => 'testCorruptedSnapshot-1677560251009,5,1677560251011.35c83130614b8300868b867ad906a41e.', STARTKEY => '5', ENDKEY => ''}] 2023-02-28 04:57:37,435 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'testCorruptedSnapshot-1677560251009' as deleted. 2023-02-28 04:57:37,435 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1677560251009","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1677560257435"}]},"ts":"9223372036854775807"} 2023-02-28 04:57:37,436 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table testCorruptedSnapshot-1677560251009 state from META 2023-02-28 04:57:37,438 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=283, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1677560251009 2023-02-28 04:57:37,439 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=283, state=SUCCESS; DeleteTableProcedure table=testCorruptedSnapshot-1677560251009 in 51 msec 2023-02-28 04:57:37,459 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-28 04:57:37,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-28 04:57:37,642 INFO [Listener at localhost/36897] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testCorruptedSnapshot-1677560251009, procId: 283 completed 2023-02-28 04:57:37,646 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1677560251009" 2023-02-28 04:57:37,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1677560251009 2023-02-28 04:57:37,649 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1677560251009" 2023-02-28 04:57:37,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42505] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1677560251009 2023-02-28 04:57:37,669 INFO [Listener at localhost/36897] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=424 (was 429), OpenFileDescriptor=790 (was 818), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=189 (was 204), ProcessCount=170 (was 170), AvailableMemoryMB=8766 (was 8872) 2023-02-28 04:57:37,670 INFO [Listener at localhost/36897] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-28 04:57:37,670 INFO [Listener at localhost/36897] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-28 04:57:37,671 DEBUG [Listener at localhost/36897] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0da5f7e5 to 127.0.0.1:54355 2023-02-28 04:57:37,671 DEBUG [Listener at localhost/36897] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:37,671 DEBUG [Listener at localhost/36897] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-28 04:57:37,671 DEBUG [Listener at localhost/36897] util.JVMClusterUtil(257): Found active master hash=1393336429, stopped=false 2023-02-28 04:57:37,671 INFO [Listener at localhost/36897] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,42505,1677560223662 2023-02-28 04:57:37,676 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:36597-0x10099497e1c0002, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-28 04:57:37,676 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-28 04:57:37,676 INFO [Listener at localhost/36897] procedure2.ProcedureExecutor(629): Stopping 2023-02-28 04:57:37,676 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:38247-0x10099497e1c0003, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-28 04:57:37,676 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-28 04:57:37,677 DEBUG [Listener at localhost/36897] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f55e535 to 127.0.0.1:54355 2023-02-28 04:57:37,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-28 04:57:37,676 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:46629-0x10099497e1c0001, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-28 04:57:37,677 DEBUG [Listener at localhost/36897] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:37,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36597-0x10099497e1c0002, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-28 04:57:37,677 INFO [Listener at localhost/36897] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46629,1677560224813' ***** 2023-02-28 04:57:37,677 INFO [Listener at localhost/36897] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-28 04:57:37,677 INFO [Listener at localhost/36897] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,36597,1677560224857' ***** 2023-02-28 04:57:37,677 INFO [Listener at localhost/36897] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-28 04:57:37,677 INFO [Listener at localhost/36897] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,38247,1677560224887' ***** 2023-02-28 04:57:37,677 INFO [Listener at localhost/36897] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-28 04:57:37,678 INFO [RS:2;jenkins-hbase4:38247] regionserver.HeapMemoryManager(220): Stopping 2023-02-28 04:57:37,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38247-0x10099497e1c0003, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-28 04:57:37,678 INFO [RS:2;jenkins-hbase4:38247] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-28 04:57:37,678 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-28 04:57:37,678 INFO [RS:2;jenkins-hbase4:38247] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-28 04:57:37,677 INFO [RS:1;jenkins-hbase4:36597] regionserver.HeapMemoryManager(220): Stopping 2023-02-28 04:57:37,678 INFO [RS:1;jenkins-hbase4:36597] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-28 04:57:37,678 INFO [RS:1;jenkins-hbase4:36597] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-28 04:57:37,678 INFO [RS:1;jenkins-hbase4:36597] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:37,678 DEBUG [RS:1;jenkins-hbase4:36597] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x76b7cd6b to 127.0.0.1:54355 2023-02-28 04:57:37,678 DEBUG [RS:1;jenkins-hbase4:36597] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:37,678 INFO [RS:1;jenkins-hbase4:36597] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,36597,1677560224857; all regions closed. 2023-02-28 04:57:37,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:46629-0x10099497e1c0001, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-28 04:57:37,677 INFO [RS:0;jenkins-hbase4:46629] regionserver.HeapMemoryManager(220): Stopping 2023-02-28 04:57:37,678 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-28 04:57:37,679 INFO [RS:0;jenkins-hbase4:46629] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-28 04:57:37,678 INFO [RS:2;jenkins-hbase4:38247] regionserver.HRegionServer(3304): Received CLOSE for befed80c35fc04cd1003e1970c424534 2023-02-28 04:57:37,679 INFO [RS:0;jenkins-hbase4:46629] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-28 04:57:37,679 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-28 04:57:37,679 INFO [RS:0;jenkins-hbase4:46629] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:37,679 DEBUG [RS:0;jenkins-hbase4:46629] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x365445cb to 127.0.0.1:54355 2023-02-28 04:57:37,679 DEBUG [RS:0;jenkins-hbase4:46629] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:37,679 INFO [RS:2;jenkins-hbase4:38247] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:37,679 INFO [RS:0;jenkins-hbase4:46629] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46629,1677560224813; all regions closed. 2023-02-28 04:57:37,679 DEBUG [RS:2;jenkins-hbase4:38247] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e3b6661 to 127.0.0.1:54355 2023-02-28 04:57:37,679 DEBUG [RS:2;jenkins-hbase4:38247] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:37,679 INFO [RS:2;jenkins-hbase4:38247] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-28 04:57:37,679 INFO [RS:2;jenkins-hbase4:38247] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-28 04:57:37,679 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing befed80c35fc04cd1003e1970c424534, disabling compactions & flushes 2023-02-28 04:57:37,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534. 2023-02-28 04:57:37,679 INFO [RS:2;jenkins-hbase4:38247] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-28 04:57:37,679 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534. 2023-02-28 04:57:37,680 INFO [RS:2;jenkins-hbase4:38247] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-28 04:57:37,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534. after waiting 0 ms 2023-02-28 04:57:37,680 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534. 2023-02-28 04:57:37,680 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing befed80c35fc04cd1003e1970c424534 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-28 04:57:37,680 INFO [RS:2;jenkins-hbase4:38247] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-02-28 04:57:37,680 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-28 04:57:37,680 DEBUG [RS:2;jenkins-hbase4:38247] regionserver.HRegionServer(1479): Online Regions={befed80c35fc04cd1003e1970c424534=hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534., 1588230740=hbase:meta,,1.1588230740} 2023-02-28 04:57:37,680 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-28 04:57:37,680 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-28 04:57:37,680 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-28 04:57:37,680 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-28 04:57:37,680 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=137.71 KB heapSize=212.48 KB 2023-02-28 04:57:37,685 DEBUG [RS:2;jenkins-hbase4:38247] regionserver.HRegionServer(1505): Waiting on 1588230740, befed80c35fc04cd1003e1970c424534 2023-02-28 04:57:37,693 DEBUG [RS:0;jenkins-hbase4:46629] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/oldWALs 2023-02-28 04:57:37,693 INFO [RS:0;jenkins-hbase4:46629] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C46629%2C1677560224813:(num 1677560226277) 2023-02-28 04:57:37,693 DEBUG [RS:0;jenkins-hbase4:46629] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:37,693 INFO [RS:0;jenkins-hbase4:46629] regionserver.LeaseManager(133): Closed leases 2023-02-28 04:57:37,694 INFO [RS:0;jenkins-hbase4:46629] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-28 04:57:37,694 INFO [RS:0;jenkins-hbase4:46629] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-28 04:57:37,694 INFO [RS:0;jenkins-hbase4:46629] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-28 04:57:37,694 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-28 04:57:37,694 INFO [RS:0;jenkins-hbase4:46629] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-28 04:57:37,695 INFO [RS:0;jenkins-hbase4:46629] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46629 2023-02-28 04:57:37,697 DEBUG [RS:1;jenkins-hbase4:36597] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/oldWALs 2023-02-28 04:57:37,697 INFO [RS:1;jenkins-hbase4:36597] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36597%2C1677560224857:(num 1677560226277) 2023-02-28 04:57:37,697 DEBUG [RS:1;jenkins-hbase4:36597] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:37,697 INFO [RS:1;jenkins-hbase4:36597] regionserver.LeaseManager(133): Closed leases 2023-02-28 04:57:37,700 INFO [RS:1;jenkins-hbase4:36597] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-28 04:57:37,701 INFO [RS:1;jenkins-hbase4:36597] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-28 04:57:37,701 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-28 04:57:37,701 INFO [RS:1;jenkins-hbase4:36597] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-28 04:57:37,701 INFO [RS:1;jenkins-hbase4:36597] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-28 04:57:37,702 INFO [RS:1;jenkins-hbase4:36597] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36597 2023-02-28 04:57:37,716 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/namespace/befed80c35fc04cd1003e1970c424534/.tmp/info/8361ebefefd7456d8e81be60a28b4c3c 2023-02-28 04:57:37,716 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=130.94 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/.tmp/info/23aba34ecc02424e8e107d4d2d9cbd42 2023-02-28 04:57:37,722 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 23aba34ecc02424e8e107d4d2d9cbd42 2023-02-28 04:57:37,722 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/namespace/befed80c35fc04cd1003e1970c424534/.tmp/info/8361ebefefd7456d8e81be60a28b4c3c as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/namespace/befed80c35fc04cd1003e1970c424534/info/8361ebefefd7456d8e81be60a28b4c3c 2023-02-28 04:57:37,725 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/namespace/befed80c35fc04cd1003e1970c424534/info/8361ebefefd7456d8e81be60a28b4c3c, entries=2, sequenceid=6, filesize=4.8 K 2023-02-28 04:57:37,726 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for befed80c35fc04cd1003e1970c424534 in 46ms, sequenceid=6, compaction requested=false 2023-02-28 04:57:37,736 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/namespace/befed80c35fc04cd1003e1970c424534/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-28 04:57:37,738 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534. 2023-02-28 04:57:37,738 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for befed80c35fc04cd1003e1970c424534: 2023-02-28 04:57:37,738 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1677560226759.befed80c35fc04cd1003e1970c424534. 2023-02-28 04:57:37,739 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.01 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/.tmp/rep_barrier/e43526b78b274e8ab1f3d8b4d97490e2 2023-02-28 04:57:37,742 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e43526b78b274e8ab1f3d8b4d97490e2 2023-02-28 04:57:37,750 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.75 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/.tmp/table/ded711b1966245de9f1f5f4477c9eb28 2023-02-28 04:57:37,752 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ded711b1966245de9f1f5f4477c9eb28 2023-02-28 04:57:37,753 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/.tmp/info/23aba34ecc02424e8e107d4d2d9cbd42 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/info/23aba34ecc02424e8e107d4d2d9cbd42 2023-02-28 04:57:37,756 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 23aba34ecc02424e8e107d4d2d9cbd42 2023-02-28 04:57:37,756 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/info/23aba34ecc02424e8e107d4d2d9cbd42, entries=64, sequenceid=307, filesize=11.8 K 2023-02-28 04:57:37,756 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/.tmp/rep_barrier/e43526b78b274e8ab1f3d8b4d97490e2 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/rep_barrier/e43526b78b274e8ab1f3d8b4d97490e2 2023-02-28 04:57:37,759 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e43526b78b274e8ab1f3d8b4d97490e2 2023-02-28 04:57:37,759 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/rep_barrier/e43526b78b274e8ab1f3d8b4d97490e2, entries=18, sequenceid=307, filesize=7.0 K 2023-02-28 04:57:37,760 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/.tmp/table/ded711b1966245de9f1f5f4477c9eb28 as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/table/ded711b1966245de9f1f5f4477c9eb28 2023-02-28 04:57:37,762 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ded711b1966245de9f1f5f4477c9eb28 2023-02-28 04:57:37,763 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/table/ded711b1966245de9f1f5f4477c9eb28, entries=22, sequenceid=307, filesize=7.0 K 2023-02-28 04:57:37,763 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 83ms, sequenceid=307, compaction requested=false 2023-02-28 04:57:37,766 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-28 04:57:37,766 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-28 04:57:37,767 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/data/hbase/meta/1588230740/recovered.edits/310.seqid, newMaxSeqId=310, maxSeqId=1 2023-02-28 04:57:37,768 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-28 04:57:37,769 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-28 04:57:37,769 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: 2023-02-28 04:57:37,769 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-28 04:57:37,771 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-28 04:57:37,788 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:38247-0x10099497e1c0003, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:37,788 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-28 04:57:37,788 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:36597-0x10099497e1c0002, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:37,788 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:46629-0x10099497e1c0001, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,46629,1677560224813 2023-02-28 04:57:37,788 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:38247-0x10099497e1c0003, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-28 04:57:37,788 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:38247-0x10099497e1c0003, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:37,788 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:36597-0x10099497e1c0002, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-28 04:57:37,788 ERROR [Listener at localhost/36897-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@59a6e3d5 rejected from java.util.concurrent.ThreadPoolExecutor@5d997c39[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-28 04:57:37,788 ERROR [Listener at localhost/36897-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7329d2c rejected from java.util.concurrent.ThreadPoolExecutor@30d1912[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-28 04:57:37,788 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:36597-0x10099497e1c0002, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:37,788 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:46629-0x10099497e1c0001, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-28 04:57:37,788 ERROR [Listener at localhost/36897-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@644b9bd6 rejected from java.util.concurrent.ThreadPoolExecutor@30d1912[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-28 04:57:37,788 ERROR [Listener at localhost/36897-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@471f7eeb rejected from java.util.concurrent.ThreadPoolExecutor@5d997c39[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-28 04:57:37,789 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:46629-0x10099497e1c0001, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36597,1677560224857 2023-02-28 04:57:37,789 ERROR [Listener at localhost/36897-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3f275c32 rejected from java.util.concurrent.ThreadPoolExecutor@5d997c39[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-28 04:57:37,789 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,36597,1677560224857] 2023-02-28 04:57:37,789 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,36597,1677560224857; numProcessing=1 2023-02-28 04:57:37,791 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,36597,1677560224857 already deleted, retry=false 2023-02-28 04:57:37,791 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,36597,1677560224857 expired; onlineServers=2 2023-02-28 04:57:37,791 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,46629,1677560224813] 2023-02-28 04:57:37,791 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,46629,1677560224813; numProcessing=2 2023-02-28 04:57:37,793 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,46629,1677560224813 already deleted, retry=false 2023-02-28 04:57:37,793 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,46629,1677560224813 expired; onlineServers=1 2023-02-28 04:57:37,885 INFO [RS:2;jenkins-hbase4:38247] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,38247,1677560224887; all regions closed. 2023-02-28 04:57:37,891 DEBUG [RS:2;jenkins-hbase4:38247] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/oldWALs 2023-02-28 04:57:37,891 INFO [RS:2;jenkins-hbase4:38247] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38247%2C1677560224887.meta:.meta(num 1677560226482) 2023-02-28 04:57:37,894 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/WALs/jenkins-hbase4.apache.org,38247,1677560224887/jenkins-hbase4.apache.org%2C38247%2C1677560224887.1677560226278 not finished, retry = 0 2023-02-28 04:57:37,996 DEBUG [RS:2;jenkins-hbase4:38247] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/oldWALs 2023-02-28 04:57:37,996 INFO [RS:2;jenkins-hbase4:38247] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38247%2C1677560224887:(num 1677560226278) 2023-02-28 04:57:37,996 DEBUG [RS:2;jenkins-hbase4:38247] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:37,996 INFO [RS:2;jenkins-hbase4:38247] regionserver.LeaseManager(133): Closed leases 2023-02-28 04:57:37,997 INFO [RS:2;jenkins-hbase4:38247] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-28 04:57:37,997 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-28 04:57:37,998 INFO [RS:2;jenkins-hbase4:38247] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38247 2023-02-28 04:57:38,000 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:38247-0x10099497e1c0003, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38247,1677560224887 2023-02-28 04:57:38,000 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-28 04:57:38,000 ERROR [Listener at localhost/36897-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1c9353d9 rejected from java.util.concurrent.ThreadPoolExecutor@18acd493[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-28 04:57:38,001 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,38247,1677560224887] 2023-02-28 04:57:38,001 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,38247,1677560224887; numProcessing=3 2023-02-28 04:57:38,003 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,38247,1677560224887 already deleted, retry=false 2023-02-28 04:57:38,003 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,38247,1677560224887 expired; onlineServers=0 2023-02-28 04:57:38,003 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,42505,1677560223662' ***** 2023-02-28 04:57:38,003 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-28 04:57:38,003 DEBUG [M:0;jenkins-hbase4:42505] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1231aa56, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-28 04:57:38,003 INFO [M:0;jenkins-hbase4:42505] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,42505,1677560223662 2023-02-28 04:57:38,004 INFO [M:0;jenkins-hbase4:42505] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,42505,1677560223662; all regions closed. 2023-02-28 04:57:38,004 DEBUG [M:0;jenkins-hbase4:42505] ipc.AbstractRpcClient(494): Stopping rpc client 2023-02-28 04:57:38,004 DEBUG [M:0;jenkins-hbase4:42505] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-28 04:57:38,004 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-28 04:57:38,004 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1677560225940] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1677560225940,5,FailOnTimeoutGroup] 2023-02-28 04:57:38,004 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1677560225940] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1677560225940,5,FailOnTimeoutGroup] 2023-02-28 04:57:38,004 DEBUG [M:0;jenkins-hbase4:42505] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-28 04:57:38,005 INFO [M:0;jenkins-hbase4:42505] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-28 04:57:38,005 INFO [M:0;jenkins-hbase4:42505] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-28 04:57:38,005 INFO [M:0;jenkins-hbase4:42505] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-28 04:57:38,005 DEBUG [M:0;jenkins-hbase4:42505] master.HMaster(1512): Stopping service threads 2023-02-28 04:57:38,005 INFO [M:0;jenkins-hbase4:42505] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-28 04:57:38,005 ERROR [M:0;jenkins-hbase4:42505] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-28 04:57:38,006 INFO [M:0;jenkins-hbase4:42505] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-28 04:57:38,006 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-28 04:57:38,007 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-28 04:57:38,007 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-28 04:57:38,007 DEBUG [M:0;jenkins-hbase4:42505] zookeeper.ZKUtil(398): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-28 04:57:38,007 WARN [M:0;jenkins-hbase4:42505] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-28 04:57:38,007 INFO [M:0;jenkins-hbase4:42505] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-28 04:57:38,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-28 04:57:38,007 INFO [M:0;jenkins-hbase4:42505] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-28 04:57:38,007 DEBUG [M:0;jenkins-hbase4:42505] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-28 04:57:38,007 INFO [M:0;jenkins-hbase4:42505] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-28 04:57:38,007 DEBUG [M:0;jenkins-hbase4:42505] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-28 04:57:38,007 DEBUG [M:0;jenkins-hbase4:42505] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-28 04:57:38,007 DEBUG [M:0;jenkins-hbase4:42505] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-28 04:57:38,007 INFO [M:0;jenkins-hbase4:42505] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=845.81 KB heapSize=1.01 MB 2023-02-28 04:57:38,019 INFO [M:0;jenkins-hbase4:42505] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=845.81 KB at sequenceid=2120 (bloomFilter=true), to=hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b119ab936fa14844af04161cace7ff8b 2023-02-28 04:57:38,021 INFO [M:0;jenkins-hbase4:42505] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b119ab936fa14844af04161cace7ff8b 2023-02-28 04:57:38,022 DEBUG [M:0;jenkins-hbase4:42505] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b119ab936fa14844af04161cace7ff8b as hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b119ab936fa14844af04161cace7ff8b 2023-02-28 04:57:38,024 INFO [M:0;jenkins-hbase4:42505] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b119ab936fa14844af04161cace7ff8b 2023-02-28 04:57:38,025 INFO [M:0;jenkins-hbase4:42505] regionserver.HStore(1080): Added hdfs://localhost:33733/user/jenkins/test-data/b78b086b-15bc-8b31-6cbc-82be6f03d92f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b119ab936fa14844af04161cace7ff8b, entries=282, sequenceid=2120, filesize=30.9 K 2023-02-28 04:57:38,025 INFO [M:0;jenkins-hbase4:42505] regionserver.HRegion(2947): Finished flush of dataSize ~845.81 KB/866113, heapSize ~1.01 MB/1062952, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 18ms, sequenceid=2120, compaction requested=false 2023-02-28 04:57:38,026 INFO [M:0;jenkins-hbase4:42505] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-28 04:57:38,026 DEBUG [M:0;jenkins-hbase4:42505] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-02-28 04:57:38,030 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-28 04:57:38,030 INFO [M:0;jenkins-hbase4:42505] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-28 04:57:38,031 INFO [M:0;jenkins-hbase4:42505] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42505 2023-02-28 04:57:38,032 DEBUG [M:0;jenkins-hbase4:42505] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,42505,1677560223662 already deleted, retry=false 2023-02-28 04:57:38,176 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-28 04:57:38,176 INFO [M:0;jenkins-hbase4:42505] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,42505,1677560223662; zookeeper connection closed. 2023-02-28 04:57:38,176 ERROR [Listener at localhost/36897-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@e4c0fca rejected from java.util.concurrent.ThreadPoolExecutor@2b6faf8c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 27] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-28 04:57:38,176 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): master:42505-0x10099497e1c0000, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-28 04:57:38,176 ERROR [Listener at localhost/36897-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@8f2e996 rejected from java.util.concurrent.ThreadPoolExecutor@2b6faf8c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 27] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-28 04:57:38,276 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:38247-0x10099497e1c0003, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-28 04:57:38,276 INFO [RS:2;jenkins-hbase4:38247] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,38247,1677560224887; zookeeper connection closed. 2023-02-28 04:57:38,276 ERROR [Listener at localhost/36897-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6a325e13 rejected from java.util.concurrent.ThreadPoolExecutor@18acd493[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-28 04:57:38,276 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:38247-0x10099497e1c0003, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-28 04:57:38,276 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6031d073] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6031d073 2023-02-28 04:57:38,276 ERROR [Listener at localhost/36897-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@12cf9c5f rejected from java.util.concurrent.ThreadPoolExecutor@18acd493[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-28 04:57:38,376 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:36597-0x10099497e1c0002, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-28 04:57:38,376 INFO [RS:1;jenkins-hbase4:36597] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,36597,1677560224857; zookeeper connection closed. 2023-02-28 04:57:38,376 ERROR [Listener at localhost/36897-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1cade3f5 rejected from java.util.concurrent.ThreadPoolExecutor@30d1912[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-28 04:57:38,376 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:36597-0x10099497e1c0002, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-28 04:57:38,376 ERROR [Listener at localhost/36897-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1e8dc261 rejected from java.util.concurrent.ThreadPoolExecutor@30d1912[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 5] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-28 04:57:38,376 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5258112] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5258112 2023-02-28 04:57:38,476 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:46629-0x10099497e1c0001, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-28 04:57:38,476 INFO [RS:0;jenkins-hbase4:46629] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46629,1677560224813; zookeeper connection closed. 2023-02-28 04:57:38,476 ERROR [Listener at localhost/36897-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4460d77a rejected from java.util.concurrent.ThreadPoolExecutor@5d997c39[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-28 04:57:38,476 DEBUG [Listener at localhost/36897-EventThread] zookeeper.ZKWatcher(600): regionserver:46629-0x10099497e1c0001, quorum=127.0.0.1:54355, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-28 04:57:38,476 ERROR [Listener at localhost/36897-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@24231187 rejected from java.util.concurrent.ThreadPoolExecutor@5d997c39[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-28 04:57:38,476 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@43ca2e25] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@43ca2e25 2023-02-28 04:57:38,477 INFO [Listener at localhost/36897] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-28 04:57:38,477 WARN [Listener at localhost/36897] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-28 04:57:38,480 INFO [Listener at localhost/36897] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-28 04:57:38,533 WARN [BP-711124759-172.31.14.131-1677560220563 heartbeating to localhost/127.0.0.1:33733] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-711124759-172.31.14.131-1677560220563 (Datanode Uuid 8c9efb49-2a4b-4a87-bb8e-d6e1442f8c08) service to localhost/127.0.0.1:33733 2023-02-28 04:57:38,535 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/cluster_b26b0118-f74c-2753-b135-766a0fc054b5/dfs/data/data5/current/BP-711124759-172.31.14.131-1677560220563] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-28 04:57:38,535 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/cluster_b26b0118-f74c-2753-b135-766a0fc054b5/dfs/data/data6/current/BP-711124759-172.31.14.131-1677560220563] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-28 04:57:38,586 WARN [Listener at localhost/36897] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-28 04:57:38,592 INFO [Listener at localhost/36897] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-28 04:57:38,696 WARN [BP-711124759-172.31.14.131-1677560220563 heartbeating to localhost/127.0.0.1:33733] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-28 04:57:38,696 WARN [BP-711124759-172.31.14.131-1677560220563 heartbeating to localhost/127.0.0.1:33733] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-711124759-172.31.14.131-1677560220563 (Datanode Uuid c977ee8b-e160-4115-8e8e-c9f50b5a113b) service to localhost/127.0.0.1:33733 2023-02-28 04:57:38,697 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/cluster_b26b0118-f74c-2753-b135-766a0fc054b5/dfs/data/data3/current/BP-711124759-172.31.14.131-1677560220563] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-28 04:57:38,697 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/cluster_b26b0118-f74c-2753-b135-766a0fc054b5/dfs/data/data4/current/BP-711124759-172.31.14.131-1677560220563] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-28 04:57:38,698 WARN [Listener at localhost/36897] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-28 04:57:38,701 INFO [Listener at localhost/36897] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-28 04:57:38,805 WARN [BP-711124759-172.31.14.131-1677560220563 heartbeating to localhost/127.0.0.1:33733] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-28 04:57:38,805 WARN [BP-711124759-172.31.14.131-1677560220563 heartbeating to localhost/127.0.0.1:33733] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-711124759-172.31.14.131-1677560220563 (Datanode Uuid ad0147c2-ff5d-4955-80a8-0012b47fa37a) service to localhost/127.0.0.1:33733 2023-02-28 04:57:38,806 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/cluster_b26b0118-f74c-2753-b135-766a0fc054b5/dfs/data/data1/current/BP-711124759-172.31.14.131-1677560220563] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-28 04:57:38,806 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/a03f14bd-380d-95e5-d9ea-3d8f35184290/cluster_b26b0118-f74c-2753-b135-766a0fc054b5/dfs/data/data2/current/BP-711124759-172.31.14.131-1677560220563] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-28 04:57:38,833 INFO [Listener at localhost/36897] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-28 04:57:38,960 INFO [Listener at localhost/36897] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-28 04:57:39,041 INFO [Listener at localhost/36897] hbase.HBaseTestingUtility(1293): Minicluster is down